Jul 17 18:33:08.130964 ip-10-0-138-51 systemd[1]: Starting Kubernetes Kubelet...
Jul 17 18:33:08.564067 ip-10-0-138-51 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 18:33:08.564067 ip-10-0-138-51 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 17 18:33:08.564067 ip-10-0-138-51 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 18:33:08.564067 ip-10-0-138-51 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 17 18:33:08.564067 ip-10-0-138-51 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 18:33:08.565784 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.565698    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 17 18:33:08.575896 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575865    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 18:33:08.575896 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575892    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 18:33:08.575896 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575897    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 18:33:08.575896 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575900    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 18:33:08.575896 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575904    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 18:33:08.576082 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575907    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 18:33:08.576082 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575911    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 18:33:08.576082 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575914    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 18:33:08.576082 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575918    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 18:33:08.576082 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575922    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 18:33:08.576082 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575925    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 18:33:08.576082 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575928    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 18:33:08.576082 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575931    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 18:33:08.576082 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575934    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 18:33:08.576082 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575937    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 18:33:08.576082 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575940    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 18:33:08.576082 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575943    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 18:33:08.576082 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575945    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 18:33:08.576082 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575948    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 18:33:08.576082 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575951    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 18:33:08.576082 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575953    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 18:33:08.576082 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575956    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 18:33:08.576082 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575958    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 18:33:08.576082 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575961    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 18:33:08.576082 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575964    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 18:33:08.576562 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575966    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 18:33:08.576562 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575969    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 18:33:08.576562 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575971    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 18:33:08.576562 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575974    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 18:33:08.576562 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575976    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 18:33:08.576562 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575979    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 18:33:08.576562 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575981    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 18:33:08.576562 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575984    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 18:33:08.576562 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575986    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 18:33:08.576562 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575989    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 18:33:08.576562 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575992    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 18:33:08.576562 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575994    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 18:33:08.576562 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.575996    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 18:33:08.576562 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576000    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 18:33:08.576562 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576003    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 18:33:08.576562 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576005    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 18:33:08.576562 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576007    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 18:33:08.576562 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576010    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 18:33:08.576562 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576012    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 18:33:08.577042 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576015    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 18:33:08.577042 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576017    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 18:33:08.577042 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576019    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 18:33:08.577042 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576022    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 18:33:08.577042 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576024    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 18:33:08.577042 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576027    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 18:33:08.577042 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576030    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 18:33:08.577042 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576033    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 18:33:08.577042 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576035    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 18:33:08.577042 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576038    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 18:33:08.577042 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576040    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 18:33:08.577042 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576042    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 18:33:08.577042 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576045    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 18:33:08.577042 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576049    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 18:33:08.577042 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576053    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 18:33:08.577042 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576056    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 18:33:08.577042 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576058    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 18:33:08.577042 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576061    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 18:33:08.577042 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576063    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 18:33:08.577503 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576066    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 18:33:08.577503 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576068    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 18:33:08.577503 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576071    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 18:33:08.577503 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576073    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 18:33:08.577503 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576076    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 18:33:08.577503 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576079    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 18:33:08.577503 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576081    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 18:33:08.577503 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576086    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 18:33:08.577503 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576089    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 18:33:08.577503 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576092    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 18:33:08.577503 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576095    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 18:33:08.577503 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576099    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 18:33:08.577503 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576103    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 18:33:08.577503 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576106    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 18:33:08.577503 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576110    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 18:33:08.577503 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576113    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 18:33:08.577503 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576116    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 18:33:08.577503 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576118    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 18:33:08.577503 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576121    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 18:33:08.577974 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576123    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 18:33:08.577974 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576128    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 18:33:08.577974 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576130    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 18:33:08.577974 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576133    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 18:33:08.577974 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576136    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 18:33:08.577974 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576138    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 18:33:08.577974 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576141    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 18:33:08.577974 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576143    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 18:33:08.577974 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576145    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 18:33:08.577974 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576148    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 18:33:08.577974 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576151    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 18:33:08.577974 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576153    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 18:33:08.577974 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576156    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 18:33:08.577974 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576158    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 18:33:08.577974 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576161    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 18:33:08.577974 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576178    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 18:33:08.577974 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.576180    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 18:33:08.578390 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578328    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 18:33:08.578390 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578337    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 18:33:08.578390 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578340    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 18:33:08.578390 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578343    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 18:33:08.578390 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578346    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 18:33:08.578390 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578348    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 18:33:08.578390 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578352    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 18:33:08.578390 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578354    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 18:33:08.578390 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578357    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 18:33:08.578390 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578361    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 18:33:08.578390 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578365    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 18:33:08.578390 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578368    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 18:33:08.578390 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578370    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 18:33:08.578390 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578373    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 18:33:08.578390 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578376    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 18:33:08.578390 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578379    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 18:33:08.578390 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578382    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 18:33:08.578390 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578385    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 18:33:08.578390 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578388    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 18:33:08.578864 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578391    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 18:33:08.578864 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578394    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 18:33:08.578864 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578397    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 18:33:08.578864 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578400    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 18:33:08.578864 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578403    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 18:33:08.578864 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578405    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 18:33:08.578864 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578409    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 18:33:08.578864 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578413    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 18:33:08.578864 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578415    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 18:33:08.578864 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578418    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 18:33:08.578864 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578420    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 18:33:08.578864 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578423    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 18:33:08.578864 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578425    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 18:33:08.578864 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578428    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 18:33:08.578864 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578430    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 18:33:08.578864 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578433    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 18:33:08.578864 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578435    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 18:33:08.578864 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578438    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 18:33:08.578864 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578440    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 18:33:08.578864 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578442    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 18:33:08.579350 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578445    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 18:33:08.579350 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578447    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 18:33:08.579350 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578450    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 18:33:08.579350 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578453    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 18:33:08.579350 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578456    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 18:33:08.579350 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578459    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 18:33:08.579350 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578462    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 18:33:08.579350 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578464    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 18:33:08.579350 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578467    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 18:33:08.579350 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578470    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 18:33:08.579350 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578472    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 18:33:08.579350 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578475    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 18:33:08.579350 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578478    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 18:33:08.579350 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578481    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 18:33:08.579350 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578484    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 18:33:08.579350 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578487    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 18:33:08.579350 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578489    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 18:33:08.579350 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578492    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 18:33:08.579350 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578494    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 18:33:08.579816 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578497    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 18:33:08.579816 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578499    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 18:33:08.579816 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578501    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 18:33:08.579816 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578504    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 18:33:08.579816 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578507    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 18:33:08.579816 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578509    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 18:33:08.579816 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578512    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 18:33:08.579816 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578515    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 18:33:08.579816 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578518    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 18:33:08.579816 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578521    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 18:33:08.579816 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578523    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 18:33:08.579816 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578526    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 18:33:08.579816 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578528    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 18:33:08.579816 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578531    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 18:33:08.579816 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578533    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 18:33:08.579816 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578535    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 18:33:08.579816 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578538    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 18:33:08.579816 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578540    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 18:33:08.579816 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578544    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 18:33:08.579816 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578546    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 18:33:08.580306 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578549    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 18:33:08.580306 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578551    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 18:33:08.580306 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578554    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 18:33:08.580306 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578557    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 18:33:08.580306 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578560    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 18:33:08.580306 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578563    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 18:33:08.580306 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578566    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 18:33:08.580306 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578568    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 18:33:08.580306 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578571    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 18:33:08.580306 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578573    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 18:33:08.580306 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578576    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 18:33:08.580306 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578578    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 18:33:08.580306 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578581    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 18:33:08.580306 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578584    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 18:33:08.580306 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578586    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 18:33:08.580306 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578589    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 18:33:08.580306 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578591    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 18:33:08.580306 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578593    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 18:33:08.580306 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578596    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 18:33:08.580306 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578598    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578601    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578991    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578996    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.578999    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579002    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579004    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579007    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579009    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579012    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579015    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579018    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579021    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579024    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579027    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579029    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579032    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579034    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579037    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579041    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579043    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 18:33:08.580837 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579046    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 18:33:08.581355 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579048    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 18:33:08.581355 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579051    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 18:33:08.581355 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579053    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 18:33:08.581355 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579056    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 18:33:08.581355 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579059    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 18:33:08.581355 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579061    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 18:33:08.581355 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579063    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 18:33:08.581355 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579066    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 18:33:08.581355 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579069    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 18:33:08.581355 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579071    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 18:33:08.581355 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579073    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 18:33:08.581355 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579076    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 18:33:08.581355 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579078    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 18:33:08.581355 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579081    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 18:33:08.581355 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579083    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 18:33:08.581355 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579085    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 18:33:08.581355 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579088    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 18:33:08.581355 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579091    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 18:33:08.581355 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579093    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 18:33:08.581355 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579096    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 18:33:08.581866 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579098    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 18:33:08.581866 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579101    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 18:33:08.581866 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579104    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 18:33:08.581866 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579107    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 18:33:08.581866 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579109    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 18:33:08.581866 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579112    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 18:33:08.581866 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579114    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 18:33:08.581866 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579117    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 18:33:08.581866 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579119    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 18:33:08.581866 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579122    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 18:33:08.581866 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579125    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 18:33:08.581866 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579127    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 18:33:08.581866 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579129    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 18:33:08.581866 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579132    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 18:33:08.581866 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579134    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 18:33:08.581866 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579137    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 18:33:08.581866 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579139    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 18:33:08.581866 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579141    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 18:33:08.581866 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579145    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 18:33:08.581866 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579149    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 18:33:08.582469 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579152    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 18:33:08.582469 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579155    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 18:33:08.582469 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579158    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 18:33:08.582469 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579161    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 18:33:08.582469 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579164    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 18:33:08.582469 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579167    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 18:33:08.582469 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579170    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 18:33:08.582469 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579172    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 18:33:08.582469 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579175    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 18:33:08.582469 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579178    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 18:33:08.582469 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579181    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 18:33:08.582469 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579185    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 18:33:08.582469 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579189    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 18:33:08.582469 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579193    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 18:33:08.582469 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579196    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 18:33:08.582469 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579199    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 18:33:08.582469 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579203    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 18:33:08.582469 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579206    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 18:33:08.582469 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579209    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 18:33:08.582940 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579211    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 18:33:08.582940 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579214    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 18:33:08.582940 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579216    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 18:33:08.582940 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579219    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 18:33:08.582940 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579221    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 18:33:08.582940 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579224    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 18:33:08.582940 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579226    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 18:33:08.582940 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579229    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 18:33:08.582940 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579231    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 18:33:08.582940 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579234    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 18:33:08.582940 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579236    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 18:33:08.582940 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579238    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 18:33:08.582940 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579241    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 18:33:08.582940 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579243    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 18:33:08.582940 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579245    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 18:33:08.582940 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579248    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 18:33:08.582940 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579250    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 18:33:08.582940 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579253    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 18:33:08.582940 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579256    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579258    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579333    2559 flags.go:64] FLAG: --address="0.0.0.0"
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579340    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579347    2559 flags.go:64] FLAG: --anonymous-auth="true"
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579352    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579358    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579361    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579366    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579370    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579374    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579376    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579380    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579383    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579386    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579389    2559 flags.go:64] FLAG: --cgroup-root=""
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579392    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579394    2559 flags.go:64] FLAG: --client-ca-file=""
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579401    2559 flags.go:64] FLAG: --cloud-provider="external"
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579404    2559 flags.go:64] FLAG: --cluster-dns="[]"
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579408    2559 flags.go:64] FLAG: --cluster-domain=""
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579410    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579414    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 17 18:33:08.583420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579417    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579420    2559 flags.go:64] FLAG: --container-log-max-files="5"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579423    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579426    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579429    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579432    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579448    2559 flags.go:64] FLAG: --contention-profiling="false"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579451    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579454    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579458    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579460    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579465    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579468    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579470    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579473    2559 flags.go:64] FLAG: --enable-load-reader="false"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579478    2559 flags.go:64] FLAG: --enable-server="true"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579481    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579485    2559 flags.go:64] FLAG: --event-burst="100"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579488    2559 flags.go:64] FLAG: --event-qps="50"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579491    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579494    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579497    2559 flags.go:64] FLAG: --eviction-hard=""
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579501    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579503    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579506    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 17 18:33:08.583989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579509    2559 flags.go:64] FLAG: --eviction-soft=""
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579512    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579530    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579534    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579537    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579540    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579542    2559 flags.go:64] FLAG: --fail-swap-on="true"
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579545    2559 flags.go:64] FLAG: --feature-gates=""
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579549    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579551    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579554    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579557    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579560    2559 flags.go:64] FLAG: --healthz-port="10248"
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579563    2559 flags.go:64] FLAG: --help="false"
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579566    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-138-51.ec2.internal"
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579569    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579572    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579575    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579578    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579581    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579584    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579587    2559 flags.go:64] FLAG: --image-service-endpoint=""
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579590    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 17 18:33:08.584593 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579593    2559 flags.go:64] FLAG: --kube-api-burst="100"
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579596    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579599    2559 flags.go:64] FLAG: --kube-api-qps="50"
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579602    2559 flags.go:64] FLAG: --kube-reserved=""
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579604    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579607    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579610    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579612    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579615    2559 flags.go:64] FLAG: --lock-file=""
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579618    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579621    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579623    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579629    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579632    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579635    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579638    2559 flags.go:64] FLAG: --logging-format="text"
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579641    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579644    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579646    2559 flags.go:64] FLAG: --manifest-url=""
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579649    2559 flags.go:64] FLAG: --manifest-url-header=""
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579653    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579656    2559 flags.go:64] FLAG: --max-open-files="1000000"
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579676    2559 flags.go:64] FLAG: --max-pods="110"
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579682    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579685    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 17 18:33:08.585194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579688    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579691    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579693    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579696    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579699    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579707    2559 flags.go:64] FLAG: --node-status-max-images="50"
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579712    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579715    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579718    2559 flags.go:64] FLAG: --pod-cidr=""
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579721    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579726    2559 flags.go:64] FLAG: --pod-manifest-path=""
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579728    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579731    2559 flags.go:64] FLAG: --pods-per-core="0"
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579734    2559 flags.go:64] FLAG: --port="10250"
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579737    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579739    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e88114c6204d3317"
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579743    2559 flags.go:64] FLAG: --qos-reserved=""
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579746    2559 flags.go:64] FLAG: --read-only-port="10255"
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579748    2559 flags.go:64] FLAG: --register-node="true"
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579751    2559 flags.go:64] FLAG: --register-with-taints=""
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579755    2559 flags.go:64] FLAG: --registry-burst="10"
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579758    2559 flags.go:64] FLAG: --registry-qps="5"
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579761    2559 flags.go:64] FLAG: --reserved-cpus=""
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579763    2559 flags.go:64] FLAG: --reserved-memory=""
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579767    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 17 18:33:08.585842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579770    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579773    2559 flags.go:64] FLAG: --rotate-certificates="false"
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579775    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579778    2559 flags.go:64] FLAG: --runonce="false"
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579781    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579785    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579788    2559 flags.go:64] FLAG: --seccomp-default="false"
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579790    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579793    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579796    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579799    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579802    2559 flags.go:64] FLAG: --storage-driver-password="root"
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579804    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579807    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579811    2559 flags.go:64] FLAG: --storage-driver-user="root"
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579814    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579817    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579820    2559 flags.go:64] FLAG: --system-cgroups=""
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579823    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579828    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579831    2559 flags.go:64] FLAG: --tls-cert-file=""
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579833    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579838    2559 flags.go:64] FLAG: --tls-min-version=""
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579841    2559 flags.go:64] FLAG: --tls-private-key-file=""
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579843    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 17 18:33:08.586439 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579846    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579849    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579852    2559 flags.go:64] FLAG: --v="2"
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579855    2559 flags.go:64] FLAG: --version="false"
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579860    2559 flags.go:64] FLAG: --vmodule=""
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579864    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.579867    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579986    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579991    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579994    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.579997    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580000    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580004    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580007    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580009    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580012    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580014    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580017    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580019    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580022    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580024    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 18:33:08.587087 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580029    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 18:33:08.587575 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580031    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 18:33:08.587575 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580034    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 18:33:08.587575 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580036    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 18:33:08.587575 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580039    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 18:33:08.587575 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580041    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 18:33:08.587575 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580044    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 18:33:08.587575 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580046    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 18:33:08.587575 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580049    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 18:33:08.587575 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580051    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 18:33:08.587575 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580054    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 18:33:08.587575 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580057    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 18:33:08.587575 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580059    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 18:33:08.587575 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580062    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 18:33:08.587575 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580064    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 18:33:08.587575 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580067    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 18:33:08.587575 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580070    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 18:33:08.587575 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580072    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 18:33:08.587575 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580075    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 18:33:08.587575 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580077    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 18:33:08.587575 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580080    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 18:33:08.588084 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580082    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 18:33:08.588084 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580084    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 18:33:08.588084 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580088    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 18:33:08.588084 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580091    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 18:33:08.588084 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580093    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 18:33:08.588084 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580095    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 18:33:08.588084 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580098    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 18:33:08.588084 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580100    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 18:33:08.588084 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580103    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 18:33:08.588084 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580105    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 18:33:08.588084 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580108    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 18:33:08.588084 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580113    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 18:33:08.588084 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580117    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 18:33:08.588084 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580119    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 18:33:08.588084 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580122    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 18:33:08.588084 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580124    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 18:33:08.588084 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580127    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 18:33:08.588084 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580129    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 18:33:08.588084 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580132    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 18:33:08.588084 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580134    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 18:33:08.588592 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580137    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 18:33:08.588592 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580139    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 18:33:08.588592 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580142    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 18:33:08.588592 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580146    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 18:33:08.588592 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580149    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 18:33:08.588592 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580152    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 18:33:08.588592 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580155    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 18:33:08.588592 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580158    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 18:33:08.588592 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580167    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 18:33:08.588592 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580170    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 18:33:08.588592 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580172    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 18:33:08.588592 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580175    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 18:33:08.588592 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580177    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 18:33:08.588592 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580180    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 18:33:08.588592 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580186    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 18:33:08.588592 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580189    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 18:33:08.588592 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580191    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 18:33:08.588592 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580194    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 18:33:08.588592 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580197    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 18:33:08.589078 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580199    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 18:33:08.589078 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580202    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 18:33:08.589078 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580204    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 18:33:08.589078 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580206    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 18:33:08.589078 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580211    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 18:33:08.589078 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580213    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 18:33:08.589078 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580216    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 18:33:08.589078 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580218    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 18:33:08.589078 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580221    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 18:33:08.589078 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580223    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 18:33:08.589078 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580226    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 18:33:08.589078 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580228    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 18:33:08.589078 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580231    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 18:33:08.589078 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580233    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 18:33:08.589078 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580236    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 18:33:08.589078 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580238    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 18:33:08.589078 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580241    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 18:33:08.589078 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580243    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 18:33:08.589078 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580246    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 18:33:08.589078 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580248    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 18:33:08.589565 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580251    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 18:33:08.589565 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580253    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 18:33:08.589565 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580258    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 18:33:08.589565 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580261    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 18:33:08.589565 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.580264    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 18:33:08.589565 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.581040    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 18:33:08.590368 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.590347    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 17 18:33:08.590398 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.590369    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 17 18:33:08.590398 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.590383    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 17 18:33:08.590398 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.590390    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 17 18:33:08.590475 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590453    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 18:33:08.590475 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590459    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 18:33:08.590475 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590462    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 18:33:08.590475 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590465    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 18:33:08.590475 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590468    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 18:33:08.590475 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590471    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 18:33:08.590475 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590473    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 18:33:08.590475 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590476    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 18:33:08.590475 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590479    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 18:33:08.590716 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590482    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 18:33:08.590716 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590484    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 18:33:08.590716 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590487    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 18:33:08.590716 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590489    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 18:33:08.590716 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590492    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 18:33:08.590716 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590494    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 18:33:08.590716 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590497    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 18:33:08.590716 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590499    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 18:33:08.590716 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590501    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 18:33:08.590716 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590504    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 18:33:08.590716 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590506    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 18:33:08.590716 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590509    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 18:33:08.590716 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590511    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 18:33:08.590716 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590514    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 18:33:08.590716 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590516    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 18:33:08.590716 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590518    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 18:33:08.590716 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590521    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 18:33:08.590716 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590524    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 18:33:08.590716 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590526    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 18:33:08.590716 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590529    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 18:33:08.591190 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590538    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 18:33:08.591190 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590540    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 18:33:08.591190 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590543    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 18:33:08.591190 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590545    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 18:33:08.591190 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590548    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 18:33:08.591190 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590550    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 18:33:08.591190 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590553    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 18:33:08.591190 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590555    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 18:33:08.591190 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590558    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 18:33:08.591190 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590560    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 18:33:08.591190 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590563    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 18:33:08.591190 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590565    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 18:33:08.591190 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590568    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 18:33:08.591190 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590571    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 18:33:08.591190 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590574    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 18:33:08.591190 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590578    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 18:33:08.591190 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590581    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 18:33:08.591190 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590584    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 18:33:08.591190 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590587    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 18:33:08.591190 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590590    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 18:33:08.591682 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590592    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 18:33:08.591682 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590595    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 18:33:08.591682 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590597    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 18:33:08.591682 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590600    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 18:33:08.591682 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590602    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 18:33:08.591682 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590605    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 18:33:08.591682 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590607    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 18:33:08.591682 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590610    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 18:33:08.591682 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590613    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 18:33:08.591682 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590617    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 18:33:08.591682 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590620    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 18:33:08.591682 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590623    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 18:33:08.591682 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590625    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 18:33:08.591682 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590634    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 18:33:08.591682 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590637    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 18:33:08.591682 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590639    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 18:33:08.591682 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590642    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 18:33:08.591682 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590644    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 18:33:08.591682 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590647    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 18:33:08.592140 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590650    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 18:33:08.592140 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590652    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 18:33:08.592140 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590655    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 18:33:08.592140 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590657    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 18:33:08.592140 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590659    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 18:33:08.592140 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590675    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 18:33:08.592140 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590678    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 18:33:08.592140 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590683    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 18:33:08.592140 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590686    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 18:33:08.592140 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590689    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 18:33:08.592140 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590692    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 18:33:08.592140 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590695    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 18:33:08.592140 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590698    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 18:33:08.592140 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590701    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 18:33:08.592140 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590703    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 18:33:08.592140 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590706    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 18:33:08.592140 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590708    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 18:33:08.592140 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590711    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 18:33:08.592140 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590713    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 18:33:08.592140 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590715    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 18:33:08.592631 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590718    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 18:33:08.592631 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590721    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 18:33:08.592631 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590723    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 18:33:08.592631 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590726    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 18:33:08.592631 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590728    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 18:33:08.592631 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590731    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 18:33:08.592631 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590734    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 18:33:08.592631 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590742    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 18:33:08.592631 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590745    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 18:33:08.592631 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590747    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 18:33:08.592631 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590750    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 18:33:08.592631 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.590754    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 18:33:08.592631 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590905    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 18:33:08.592631 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590909    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 18:33:08.592631 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590912    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 18:33:08.593003 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590915    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 18:33:08.593003 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590919    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 18:33:08.593003 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590922    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 18:33:08.593003 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590926    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 18:33:08.593003 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590930    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 18:33:08.593003 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590933    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 18:33:08.593003 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590936    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 18:33:08.593003 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590938    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 18:33:08.593003 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590942    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 18:33:08.593003 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590945    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 18:33:08.593003 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590948    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 18:33:08.593003 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590951    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 18:33:08.593003 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590953    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 18:33:08.593003 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590956    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 18:33:08.593003 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590958    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 18:33:08.593003 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590961    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 18:33:08.593003 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590963    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 18:33:08.593003 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590966    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 18:33:08.593003 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590970    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 18:33:08.593452 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590972    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 18:33:08.593452 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590975    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 18:33:08.593452 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590977    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 18:33:08.593452 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590980    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 18:33:08.593452 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590982    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 18:33:08.593452 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590985    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 18:33:08.593452 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590993    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 18:33:08.593452 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590995    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 18:33:08.593452 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.590998    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 18:33:08.593452 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591000    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 18:33:08.593452 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591003    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 18:33:08.593452 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591005    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 18:33:08.593452 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591007    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 18:33:08.593452 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591010    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 18:33:08.593452 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591012    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 18:33:08.593452 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591015    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 18:33:08.593452 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591017    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 18:33:08.593452 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591020    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 18:33:08.593452 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591023    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 18:33:08.593452 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591025    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 18:33:08.593961 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591028    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 18:33:08.593961 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591030    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 18:33:08.593961 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591033    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 18:33:08.593961 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591035    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 18:33:08.593961 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591038    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 18:33:08.593961 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591040    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 18:33:08.593961 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591043    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 18:33:08.593961 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591045    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 18:33:08.593961 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591048    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 18:33:08.593961 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591050    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 18:33:08.593961 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591052    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 18:33:08.593961 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591055    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 18:33:08.593961 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591057    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 18:33:08.593961 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591059    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 18:33:08.593961 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591062    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 18:33:08.593961 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591064    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 18:33:08.593961 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591067    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 18:33:08.593961 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591069    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 18:33:08.593961 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591072    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 18:33:08.593961 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591081    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 18:33:08.594438 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591083    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 18:33:08.594438 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591086    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 18:33:08.594438 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591088    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 18:33:08.594438 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591090    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 18:33:08.594438 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591093    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 18:33:08.594438 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591095    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 18:33:08.594438 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591097    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 18:33:08.594438 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591100    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 18:33:08.594438 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591103    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 18:33:08.594438 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591105    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 18:33:08.594438 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591108    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 18:33:08.594438 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591110    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 18:33:08.594438 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591113    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 18:33:08.594438 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591116    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 18:33:08.594438 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591118    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 18:33:08.594438 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591120    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 18:33:08.594438 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591123    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 18:33:08.594438 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591125    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 18:33:08.594438 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591128    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 18:33:08.594438 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591130    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 18:33:08.594929 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591133    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 18:33:08.594929 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591135    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 18:33:08.594929 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591138    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 18:33:08.594929 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591140    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 18:33:08.594929 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591143    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 18:33:08.594929 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591145    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 18:33:08.594929 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591147    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 18:33:08.594929 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591150    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 18:33:08.594929 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591152    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 18:33:08.594929 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591155    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 18:33:08.594929 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591157    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 18:33:08.594929 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591160    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 18:33:08.594929 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591167    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 18:33:08.594929 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591170    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 18:33:08.594929 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591173    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 18:33:08.594929 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591175    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 18:33:08.594929 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:08.591177    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 18:33:08.595331 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.591182    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 18:33:08.595331 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.591308    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 17 18:33:08.595331 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.593323    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 17 18:33:08.595331 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.594584    2559 server.go:1019] "Starting client certificate rotation"
Jul 17 18:33:08.595331 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.594723    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 18:33:08.595331 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.594752    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 18:33:08.597217 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.597204    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 18:33:08.602368 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.602351    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 18:33:08.610293 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.610277    2559 log.go:25] "Validated CRI v1 runtime API"
Jul 17 18:33:08.614278 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.614261    2559 log.go:25] "Validated CRI v1 image API"
Jul 17 18:33:08.617255 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.617238    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 17 18:33:08.619526 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.619508    2559 fs.go:135] Filesystem UUIDs: map[3140670c-c912-479c-9620-9ad67247acfa:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7c6a5b63-58e4-47d8-923e-a3269047d70b:/dev/nvme0n1p4]
Jul 17 18:33:08.619575 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.619526    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 17 18:33:08.624940 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.624828    2559 manager.go:216] Machine: {Timestamp:2026-07-17 18:33:08.623082546 +0000 UTC m=+0.366232682 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3124771 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b864c5bef1d9d62ec2f4cd618fbfb SystemUUID:ec2b864c-5bef-1d9d-62ec-2f4cd618fbfb BootID:8d993a32-4207-4d8c-8049-7543f26e97e7 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:df:e2:26:93:4f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:df:e2:26:93:4f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:b3:e9:83:76:63 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 17 18:33:08.624940 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.624937    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 17 18:33:08.625057 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.625014    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 17 18:33:08.627295 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.627268    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 17 18:33:08.627426 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.627298    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-51.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 17 18:33:08.628055 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.628046    2559 topology_manager.go:138] "Creating topology manager with none policy"
Jul 17 18:33:08.628093 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.628057    2559 container_manager_linux.go:309] "Creating device plugin manager"
Jul 17 18:33:08.628093 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.628065    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 18:33:08.628792 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.628783    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 18:33:08.629551 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.629533    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 18:33:08.630018 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.630006    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 17 18:33:08.632792 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.632781    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 18:33:08.632908 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.632898    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 17 18:33:08.632951 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.632931    2559 kubelet.go:486] "Attempting to sync node with API server"
Jul 17 18:33:08.632951 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.632945    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 17 18:33:08.633004 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.632968    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 17 18:33:08.633004 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.632982    2559 kubelet.go:398] "Adding apiserver pod source"
Jul 17 18:33:08.633004 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.632996    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 17 18:33:08.633447 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.633434    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 18:33:08.633487 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.633463    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 18:33:08.635714 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.635696    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 17 18:33:08.637379 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.637365    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 17 18:33:08.637426 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.637397    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 17 18:33:08.639458 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.639442    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 17 18:33:08.639494 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.639464    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 17 18:33:08.639494 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.639471    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 17 18:33:08.639494 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.639478    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 17 18:33:08.639494 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.639487    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 17 18:33:08.639494 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.639493    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 17 18:33:08.639646 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.639501    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 17 18:33:08.639646 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.639506    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 17 18:33:08.639646 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.639512    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 17 18:33:08.639646 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.639517    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 17 18:33:08.639646 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.639531    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 17 18:33:08.639646 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.639536    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 17 18:33:08.639646 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.639564    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 17 18:33:08.639646 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.639570    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 17 18:33:08.642598 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.642585    2559 server.go:1268] "Started kubelet"
Jul 17 18:33:08.642717 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.642691    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 17 18:33:08.642888 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.642793    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 17 18:33:08.643018 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.643005    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 17 18:33:08.643396 ip-10-0-138-51 systemd[1]: Started Kubernetes Kubelet.
Jul 17 18:33:08.644747 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.644728    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 17 18:33:08.645245 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.645230    2559 server.go:310] "Adding debug handlers to kubelet server"
Jul 17 18:33:08.648316 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:08.648252    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 17 18:33:08.648402 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.648334    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-51.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 17 18:33:08.648483 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:08.648456    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-51.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 17 18:33:08.648874 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:08.648845    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 17 18:33:08.648935 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:08.647924    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-51.ec2.internal.18c3276431158e4d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-51.ec2.internal,UID:ip-10-0-138-51.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-51.ec2.internal,},FirstTimestamp:2026-07-17 18:33:08.642557517 +0000 UTC m=+0.385707654,LastTimestamp:2026-07-17 18:33:08.642557517 +0000 UTC m=+0.385707654,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-51.ec2.internal,}"
Jul 17 18:33:08.652514 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.652429    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 17 18:33:08.652514 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.652440    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 17 18:33:08.652801 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:08.652778    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-51.ec2.internal\" not found"
Jul 17 18:33:08.653256 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.653174    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 17 18:33:08.653256 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.653191    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 17 18:33:08.653256 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.653216    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 17 18:33:08.653441 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.653427    2559 reconstruct.go:97] "Volume reconstruction finished"
Jul 17 18:33:08.653441 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.653442    2559 reconciler.go:29] "Reconciler: start to sync state"
Jul 17 18:33:08.653552 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.653533    2559 factory.go:55] Registering systemd factory
Jul 17 18:33:08.653552 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.653553    2559 factory.go:223] Registration of the systemd container factory successfully
Jul 17 18:33:08.654353 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.654319    2559 factory.go:153] Registering CRI-O factory
Jul 17 18:33:08.654353 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.654355    2559 factory.go:223] Registration of the crio container factory successfully
Jul 17 18:33:08.654483 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.654414    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 17 18:33:08.654483 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.654438    2559 factory.go:103] Registering Raw factory
Jul 17 18:33:08.654620 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.654487    2559 manager.go:1196] Started watching for new ooms in manager
Jul 17 18:33:08.655823 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:08.655774    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 17 18:33:08.656205 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:08.656159    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-51.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 17 18:33:08.656280 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.656207    2559 manager.go:349] Starting recovery of all containers
Jul 17 18:33:08.667511 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.667347    2559 manager.go:354] Recovery completed
Jul 17 18:33:08.671507 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.671494    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:33:08.673342 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.673325    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:33:08.673429 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.673365    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:33:08.673429 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.673376    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:33:08.673978 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.673962    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 17 18:33:08.674047 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.673978    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 17 18:33:08.674047 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.673995    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 18:33:08.674047 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.673997    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:33:08.674681 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.674651    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:33:08.674764 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.674724    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:33:08.674764 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.674744    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:33:08.675335 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.675316    2559 policy_none.go:49] "None policy: Start"
Jul 17 18:33:08.675335 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.675336    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 17 18:33:08.675440 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.675347    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 17 18:33:08.676239 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.676228    2559 policy_none.go:47] "Start"
Jul 17 18:33:08.685554 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:08.685490    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-51.ec2.internal.18c3276432eb70af  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-51.ec2.internal,UID:ip-10-0-138-51.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-51.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-51.ec2.internal,},FirstTimestamp:2026-07-17 18:33:08.673351855 +0000 UTC m=+0.416501994,LastTimestamp:2026-07-17 18:33:08.673351855 +0000 UTC m=+0.416501994,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-51.ec2.internal,}"
Jul 17 18:33:08.686812 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.686792    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c667q"
Jul 17 18:33:08.693988 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:08.693907    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-51.ec2.internal.18c3276432ebb8e1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-51.ec2.internal,UID:ip-10-0-138-51.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-51.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-51.ec2.internal,},FirstTimestamp:2026-07-17 18:33:08.673370337 +0000 UTC m=+0.416520474,LastTimestamp:2026-07-17 18:33:08.673370337 +0000 UTC m=+0.416520474,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-51.ec2.internal,}"
Jul 17 18:33:08.698595 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.698578    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c667q"
Jul 17 18:33:08.710201 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.710187    2559 manager.go:337] "Starting Device Plugin manager"
Jul 17 18:33:08.710295 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:08.710216    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 17 18:33:08.710295 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.710224    2559 server.go:87] "Starting device plugin registration server"
Jul 17 18:33:08.710488 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.710477    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 17 18:33:08.710531 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.710491    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 17 18:33:08.710614 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.710601    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 17 18:33:08.710716 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.710695    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 17 18:33:08.710716 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.710708    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 17 18:33:08.711201 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:08.711175    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 17 18:33:08.711276 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:08.711217    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-51.ec2.internal\" not found"
Jul 17 18:33:08.811420 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.811377    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:33:08.812873 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.812654    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:33:08.812873 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.812696    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:33:08.812873 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.812707    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:33:08.812873 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.812762    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-51.ec2.internal"
Jul 17 18:33:08.815824 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.815776    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 17 18:33:08.817195 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.817178    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 17 18:33:08.817298 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.817213    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 17 18:33:08.817298 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.817238    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 17 18:33:08.817298 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:08.817273    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 17 18:33:08.821505 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.821485    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 18:33:08.823687 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.823659    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-51.ec2.internal"
Jul 17 18:33:08.823734 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:08.823695    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-51.ec2.internal\": node \"ip-10-0-138-51.ec2.internal\" not found"
Jul 17 18:33:08.917384 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.917330    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-51.ec2.internal"]
Jul 17 18:33:08.917510 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.917423    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:33:08.918318 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.918298    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:33:08.918377 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.918328    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:33:08.918377 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.918338    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:33:08.919477 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.919464    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:33:08.919639 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.919625    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal"
Jul 17 18:33:08.919696 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.919691    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:33:08.920171 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.920149    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:33:08.920216 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.920169    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:33:08.920216 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.920192    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:33:08.920216 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.920201    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:33:08.920312 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.920174    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:33:08.920312 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.920265    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:33:08.921376 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.921343    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-51.ec2.internal"
Jul 17 18:33:08.921460 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.921383    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:33:08.922065 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.922051    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:33:08.922132 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.922075    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:33:08.922132 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.922087    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:33:08.941987 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:08.941971    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-51.ec2.internal\" not found"
Jul 17 18:33:08.954234 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:08.954208    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-51.ec2.internal\" not found" node="ip-10-0-138-51.ec2.internal"
Jul 17 18:33:08.954924 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.954904    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/68a4b3cbdf87f36c033c410a5f90d655-config\") pod \"kube-apiserver-proxy-ip-10-0-138-51.ec2.internal\" (UID: \"68a4b3cbdf87f36c033c410a5f90d655\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-51.ec2.internal"
Jul 17 18:33:08.955003 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.954938    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/80efdace1bee045fa6ee7c757931d822-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal\" (UID: \"80efdace1bee045fa6ee7c757931d822\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal"
Jul 17 18:33:08.955003 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:08.954966    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80efdace1bee045fa6ee7c757931d822-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal\" (UID: \"80efdace1bee045fa6ee7c757931d822\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal"
Jul 17 18:33:08.958552 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:08.958534    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-51.ec2.internal\" not found" node="ip-10-0-138-51.ec2.internal"
Jul 17 18:33:09.042831 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:09.042807    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-51.ec2.internal\" not found"
Jul 17 18:33:09.055202 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:09.055184    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/80efdace1bee045fa6ee7c757931d822-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal\" (UID: \"80efdace1bee045fa6ee7c757931d822\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal"
Jul 17 18:33:09.055257 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:09.055209    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80efdace1bee045fa6ee7c757931d822-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal\" (UID: \"80efdace1bee045fa6ee7c757931d822\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal"
Jul 17 18:33:09.055257 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:09.055226    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/68a4b3cbdf87f36c033c410a5f90d655-config\") pod \"kube-apiserver-proxy-ip-10-0-138-51.ec2.internal\" (UID: \"68a4b3cbdf87f36c033c410a5f90d655\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-51.ec2.internal"
Jul 17 18:33:09.055332 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:09.055274    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/68a4b3cbdf87f36c033c410a5f90d655-config\") pod \"kube-apiserver-proxy-ip-10-0-138-51.ec2.internal\" (UID: \"68a4b3cbdf87f36c033c410a5f90d655\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-51.ec2.internal"
Jul 17 18:33:09.055332 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:09.055281    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/80efdace1bee045fa6ee7c757931d822-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal\" (UID: \"80efdace1bee045fa6ee7c757931d822\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal"
Jul 17 18:33:09.055332 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:09.055308    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80efdace1bee045fa6ee7c757931d822-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal\" (UID: \"80efdace1bee045fa6ee7c757931d822\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal"
Jul 17 18:33:09.142984 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:09.142919    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-51.ec2.internal\" not found"
Jul 17 18:33:09.243908 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:09.243856    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-51.ec2.internal\" not found"
Jul 17 18:33:09.257311 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:09.257283    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal"
Jul 17 18:33:09.261397 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:09.261134    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-51.ec2.internal"
Jul 17 18:33:10.645872 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:10.645837    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-51.ec2.internal\" not found"
Jul 17 18:33:10.746429 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:10.746355    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-51.ec2.internal\" not found"
Jul 17 18:33:10.846988 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:10.846951    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-51.ec2.internal\" not found"
Jul 17 18:33:10.895434 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:10.895397    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 17 18:33:10.895883 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:10.895531    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 18:33:10.947862 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:10.947829    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-51.ec2.internal\" not found"
Jul 17 18:33:11.004263 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.004192    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 18:28:09 +0000 UTC" deadline="2027-12-21 08:06:21.347661372 +0000 UTC"
Jul 17 18:33:11.004263 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.004221    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12517h33m10.343443864s"
Jul 17 18:33:11.048585 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:11.048532    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-51.ec2.internal\" not found"
Jul 17 18:33:11.056856 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.056831    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:33:11.058583 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.058565    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:33:11.058675 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.058597    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:33:11.058675 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.058608    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:33:11.115025 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:11.114979    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a4b3cbdf87f36c033c410a5f90d655.slice/crio-036172ce2a4a4ef75ff3d3795ed94da7d9631f2053a89b15bc9756649ad025a8 WatchSource:0}: Error finding container 036172ce2a4a4ef75ff3d3795ed94da7d9631f2053a89b15bc9756649ad025a8: Status 404 returned error can't find the container with id 036172ce2a4a4ef75ff3d3795ed94da7d9631f2053a89b15bc9756649ad025a8
Jul 17 18:33:11.115465 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:11.115446    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80efdace1bee045fa6ee7c757931d822.slice/crio-244f3dd8b17dd8bf14fc67e7a0b4b16af8837b33a7868240bd070eb31807e0f6 WatchSource:0}: Error finding container 244f3dd8b17dd8bf14fc67e7a0b4b16af8837b33a7868240bd070eb31807e0f6: Status 404 returned error can't find the container with id 244f3dd8b17dd8bf14fc67e7a0b4b16af8837b33a7868240bd070eb31807e0f6
Jul 17 18:33:11.120040 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.120019    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 18:33:11.122236 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.122188    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-51.ec2.internal" event={"ID":"68a4b3cbdf87f36c033c410a5f90d655","Type":"ContainerStarted","Data":"036172ce2a4a4ef75ff3d3795ed94da7d9631f2053a89b15bc9756649ad025a8"}
Jul 17 18:33:11.123437 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.123414    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal" event={"ID":"80efdace1bee045fa6ee7c757931d822","Type":"ContainerStarted","Data":"244f3dd8b17dd8bf14fc67e7a0b4b16af8837b33a7868240bd070eb31807e0f6"}
Jul 17 18:33:11.149358 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:11.149328    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-51.ec2.internal\" not found"
Jul 17 18:33:11.213095 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.213067    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 18:33:11.255385 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.255320    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal"
Jul 17 18:33:11.285804 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.285778    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 18:33:11.291073 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.291054    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-51.ec2.internal"
Jul 17 18:33:11.292488 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.292473    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 18:33:11.315407 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.315384    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 18:33:11.437825 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.437798    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 18:33:11.701687 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.701605    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 18:33:11.936469 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.936437    2559 apiserver.go:52] "Watching apiserver"
Jul 17 18:33:11.944443 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.944422    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 17 18:33:11.946214 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.946180    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz","openshift-dns/node-resolver-njrqr","openshift-image-registry/node-ca-ztfrm","openshift-multus/multus-58rmc","openshift-multus/network-metrics-daemon-tt279","openshift-cluster-node-tuning-operator/tuned-nd8nj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal","openshift-multus/multus-additional-cni-plugins-cnj2q","openshift-network-diagnostics/network-check-target-m6ftw","openshift-network-operator/iptables-alerter-2pb8h","openshift-ovn-kubernetes/ovnkube-node-xhn5c","kube-system/global-pull-secret-syncer-5nws8","kube-system/konnectivity-agent-vtswf","kube-system/kube-apiserver-proxy-ip-10-0-138-51.ec2.internal"]
Jul 17 18:33:11.947686 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.947659    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2pb8h"
Jul 17 18:33:11.949442 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.949275    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vtswf"
Jul 17 18:33:11.951400 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.950751    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ztfrm"
Jul 17 18:33:11.951894 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.951843    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-58rmc"
Jul 17 18:33:11.952989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.952969    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:11.953081 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:11.953038    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt279" podUID="f9fc2de6-adf3-45c0-952a-e317f362aa51"
Jul 17 18:33:11.955374 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.954255    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:11.955374 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.954619    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 17 18:33:11.955814 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.955792    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:11.955903 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:11.955863    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6ftw" podUID="b6eaddd6-5198-4d18-bd28-bd1d31df6994"
Jul 17 18:33:11.956963 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.956944    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:11.965625 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.964624    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 17 18:33:11.965625 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.964707    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k858s\""
Jul 17 18:33:11.965625 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.964627    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 17 18:33:11.965625 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.964936    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 17 18:33:11.965625 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.965030    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-p2qql\""
Jul 17 18:33:11.965625 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.965074    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 17 18:33:11.965625 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.965144    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 17 18:33:11.965625 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.965247    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 17 18:33:11.965625 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.965255    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-njrqr"
Jul 17 18:33:11.966827 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.966223    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 17 18:33:11.966827 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.966364    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 17 18:33:11.966827 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.966413    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 17 18:33:11.966827 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.966422    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-etc-sysctl-d\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:11.966827 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.966466    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78304550-dac5-44f4-91e0-58503470a03a-etc-tuned\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:11.966827 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.966508    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dltsf\" (UniqueName: \"kubernetes.io/projected/78304550-dac5-44f4-91e0-58503470a03a-kube-api-access-dltsf\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:11.966827 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.966580    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 17 18:33:11.966827 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.966595    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-etc-kubernetes\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:11.966827 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.966686    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd0f1bed-d043-4fbd-90d0-f3dfb22292de-iptables-alerter-script\") pod \"iptables-alerter-2pb8h\" (UID: \"bd0f1bed-d043-4fbd-90d0-f3dfb22292de\") " pod="openshift-network-operator/iptables-alerter-2pb8h"
Jul 17 18:33:11.966827 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.966730    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:11.966827 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.966742    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-etc-kubernetes\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:11.967333 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.966866    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-lib-modules\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:11.967920 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.967894    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-host-var-lib-cni-multus\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:11.968015 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.967941    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpdgv\" (UniqueName: \"kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv\") pod \"network-check-target-m6ftw\" (UID: \"b6eaddd6-5198-4d18-bd28-bd1d31df6994\") " pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:11.968015 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.967969    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-cni-binary-copy\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:11.968015 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.967993    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-multus-socket-dir-parent\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:11.968930 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968360    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-etc-sysctl-conf\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:11.968930 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968412    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-sys\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:11.968930 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968443    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-host-var-lib-kubelet\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:11.968930 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968467    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-multus-daemon-config\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:11.968930 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968494    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xljnf\" (UniqueName: \"kubernetes.io/projected/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-kube-api-access-xljnf\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:11.968930 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968520    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d33d089a-3e2f-4edd-a459-0303cf90f1af-host\") pod \"node-ca-ztfrm\" (UID: \"d33d089a-3e2f-4edd-a459-0303cf90f1af\") " pod="openshift-image-registry/node-ca-ztfrm"
Jul 17 18:33:11.968930 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968540    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-etc-systemd\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:11.968930 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968559    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-run\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:11.968930 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968578    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-var-lib-kubelet\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:11.968930 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968599    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-host\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:11.968930 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968617    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-os-release\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:11.968930 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968637    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58ndx\" (UniqueName: \"kubernetes.io/projected/f9fc2de6-adf3-45c0-952a-e317f362aa51-kube-api-access-58ndx\") pod \"network-metrics-daemon-tt279\" (UID: \"f9fc2de6-adf3-45c0-952a-e317f362aa51\") " pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:11.968930 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968678    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e35f12b9-1b2e-474f-8261-ec3e25a25775-agent-certs\") pod \"konnectivity-agent-vtswf\" (UID: \"e35f12b9-1b2e-474f-8261-ec3e25a25775\") " pod="kube-system/konnectivity-agent-vtswf"
Jul 17 18:33:11.968930 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968698    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-host-run-netns\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:11.968930 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968722    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs\") pod \"network-metrics-daemon-tt279\" (UID: \"f9fc2de6-adf3-45c0-952a-e317f362aa51\") " pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:11.968930 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968744    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frsgq\" (UniqueName: \"kubernetes.io/projected/bd0f1bed-d043-4fbd-90d0-f3dfb22292de-kube-api-access-frsgq\") pod \"iptables-alerter-2pb8h\" (UID: \"bd0f1bed-d043-4fbd-90d0-f3dfb22292de\") " pod="openshift-network-operator/iptables-alerter-2pb8h"
Jul 17 18:33:11.968930 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968765    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78304550-dac5-44f4-91e0-58503470a03a-tmp\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:11.969808 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968790    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd0f1bed-d043-4fbd-90d0-f3dfb22292de-host-slash\") pod \"iptables-alerter-2pb8h\" (UID: \"bd0f1bed-d043-4fbd-90d0-f3dfb22292de\") " pod="openshift-network-operator/iptables-alerter-2pb8h"
Jul 17 18:33:11.969808 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968824    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-multus-cni-dir\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:11.969808 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968825    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-plbvz\""
Jul 17 18:33:11.969808 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968851    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-host-run-k8s-cni-cncf-io\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:11.969808 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968872    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-hostroot\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:11.969808 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968899    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-multus-conf-dir\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:11.969808 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968940    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-etc-modprobe-d\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:11.969808 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968961    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-etc-sysconfig\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:11.969808 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.968983    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-system-cni-dir\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:11.969808 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.969002    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-cnibin\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:11.969808 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.969054    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-host-var-lib-cni-bin\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:11.969808 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.969084    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-host-run-multus-certs\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:11.969808 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.969120    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e35f12b9-1b2e-474f-8261-ec3e25a25775-konnectivity-ca\") pod \"konnectivity-agent-vtswf\" (UID: \"e35f12b9-1b2e-474f-8261-ec3e25a25775\") " pod="kube-system/konnectivity-agent-vtswf"
Jul 17 18:33:11.969808 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.969140    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d33d089a-3e2f-4edd-a459-0303cf90f1af-serviceca\") pod \"node-ca-ztfrm\" (UID: \"d33d089a-3e2f-4edd-a459-0303cf90f1af\") " pod="openshift-image-registry/node-ca-ztfrm"
Jul 17 18:33:11.969808 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.969165    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hccv5\" (UniqueName: \"kubernetes.io/projected/d33d089a-3e2f-4edd-a459-0303cf90f1af-kube-api-access-hccv5\") pod \"node-ca-ztfrm\" (UID: \"d33d089a-3e2f-4edd-a459-0303cf90f1af\") " pod="openshift-image-registry/node-ca-ztfrm"
Jul 17 18:33:11.969808 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.969812    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:11.970544 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.969959    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dlzlv\""
Jul 17 18:33:11.970544 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.970255    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 17 18:33:11.970544 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.970514    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 17 18:33:11.971965 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.971152    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:11.971965 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:11.971215    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nws8" podUID="1f8b093e-7ab9-406d-8a0e-c79e694cb205"
Jul 17 18:33:11.985392 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:11.985374    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 17 18:33:12.015881 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.015862    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 17 18:33:12.015969 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.015921    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 17 18:33:12.016036 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.016014    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 17 18:33:12.016098 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.016065    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 17 18:33:12.016160 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.015864    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 17 18:33:12.016222 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.016210    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 17 18:33:12.016346 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.015867    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 17 18:33:12.016477 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.016462    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 17 18:33:12.016727 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.016710    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 17 18:33:12.016811 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.016796    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mhxkn\""
Jul 17 18:33:12.017133 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.017118    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2znwt\""
Jul 17 18:33:12.017253 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.017186    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m8m4t\""
Jul 17 18:33:12.017329 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.017272    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-89dk4\""
Jul 17 18:33:12.017558 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.017535    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 17 18:33:12.017620 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.017565    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 17 18:33:12.017670 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.017628    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 17 18:33:12.017670 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.017629    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hxr6b\""
Jul 17 18:33:12.018608 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.018593    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 17 18:33:12.020663 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.020644    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 17 18:33:12.033181 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.033161    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 18:33:12.053718 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.053692    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lwz9s"
Jul 17 18:33:12.056529 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.056511    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 17 18:33:12.069962 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.069934    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-etc-modprobe-d\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.070076 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.069977    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-etc-sysconfig\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.070076 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.069999    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-system-cni-dir\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.070179 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070105    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-system-cni-dir\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.070179 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070132    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-etc-modprobe-d\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.070179 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070162    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-etc-sysconfig\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.070344 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070179    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-cnibin\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.070344 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070215    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-host-var-lib-cni-bin\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.070344 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070253    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-host-run-multus-certs\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.070344 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070277    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hccv5\" (UniqueName: \"kubernetes.io/projected/d33d089a-3e2f-4edd-a459-0303cf90f1af-kube-api-access-hccv5\") pod \"node-ca-ztfrm\" (UID: \"d33d089a-3e2f-4edd-a459-0303cf90f1af\") " pod="openshift-image-registry/node-ca-ztfrm"
Jul 17 18:33:12.070344 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070319    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78304550-dac5-44f4-91e0-58503470a03a-etc-tuned\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.070574 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070352    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.070574 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070383    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/486589ec-aa0a-497e-88e6-79e52df71588-cnibin\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.070574 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070406    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dltsf\" (UniqueName: \"kubernetes.io/projected/78304550-dac5-44f4-91e0-58503470a03a-kube-api-access-dltsf\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.070574 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070432    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/486589ec-aa0a-497e-88e6-79e52df71588-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.070574 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070459    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd0f1bed-d043-4fbd-90d0-f3dfb22292de-iptables-alerter-script\") pod \"iptables-alerter-2pb8h\" (UID: \"bd0f1bed-d043-4fbd-90d0-f3dfb22292de\") " pod="openshift-network-operator/iptables-alerter-2pb8h"
Jul 17 18:33:12.070574 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070499    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-lib-modules\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.070574 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070522    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a1608b8-a53a-480d-965d-8ab7e3ec8f22-hosts-file\") pod \"node-resolver-njrqr\" (UID: \"3a1608b8-a53a-480d-965d-8ab7e3ec8f22\") " pod="openshift-dns/node-resolver-njrqr"
Jul 17 18:33:12.070574 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070545    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1eb72ece-f355-4599-b236-299ac5dce970-device-dir\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.070883 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070571    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb7l2\" (UniqueName: \"kubernetes.io/projected/486589ec-aa0a-497e-88e6-79e52df71588-kube-api-access-bb7l2\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.070883 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070605    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret\") pod \"global-pull-secret-syncer-5nws8\" (UID: \"1f8b093e-7ab9-406d-8a0e-c79e694cb205\") " pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:12.070883 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070632    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-host-var-lib-cni-multus\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.070883 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070659    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpdgv\" (UniqueName: \"kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv\") pod \"network-check-target-m6ftw\" (UID: \"b6eaddd6-5198-4d18-bd28-bd1d31df6994\") " pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:12.070883 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070688    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-cni-binary-copy\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.070883 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070715    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-multus-socket-dir-parent\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.070883 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070737    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-sys\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.070883 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070759    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-kubelet\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.070883 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070784    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-host-var-lib-kubelet\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.070883 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070842    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-host-var-lib-kubelet\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.071249 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.070887    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-multus-daemon-config\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.071249 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071105    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-cnibin\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.071249 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071149    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-host-var-lib-cni-bin\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.071249 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071205    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-multus-socket-dir-parent\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.071249 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071242    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-sys\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.071466 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071355    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-lib-modules\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.071466 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071374    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d33d089a-3e2f-4edd-a459-0303cf90f1af-host\") pod \"node-ca-ztfrm\" (UID: \"d33d089a-3e2f-4edd-a459-0303cf90f1af\") " pod="openshift-image-registry/node-ca-ztfrm"
Jul 17 18:33:12.071466 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071404    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-host-run-multus-certs\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.071466 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071405    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-etc-systemd\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.071466 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071444    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-run\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.071607 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071482    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-var-lib-kubelet\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.071607 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071512    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1eb72ece-f355-4599-b236-299ac5dce970-etc-selinux\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.071607 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071502    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-etc-systemd\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.071607 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071539    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-slash\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.071607 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071581    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-run\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.071815 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071626    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-var-lib-kubelet\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.071815 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071643    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-host-var-lib-cni-multus\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.071815 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071655    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-os-release\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.071815 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071686    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-multus-daemon-config\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.071815 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071695    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d33d089a-3e2f-4edd-a459-0303cf90f1af-host\") pod \"node-ca-ztfrm\" (UID: \"d33d089a-3e2f-4edd-a459-0303cf90f1af\") " pod="openshift-image-registry/node-ca-ztfrm"
Jul 17 18:33:12.071815 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071700    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-cni-binary-copy\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.071815 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071704    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd0f1bed-d043-4fbd-90d0-f3dfb22292de-iptables-alerter-script\") pod \"iptables-alerter-2pb8h\" (UID: \"bd0f1bed-d043-4fbd-90d0-f3dfb22292de\") " pod="openshift-network-operator/iptables-alerter-2pb8h"
Jul 17 18:33:12.071815 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071761    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 17 18:33:12.072144 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071816    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58ndx\" (UniqueName: \"kubernetes.io/projected/f9fc2de6-adf3-45c0-952a-e317f362aa51-kube-api-access-58ndx\") pod \"network-metrics-daemon-tt279\" (UID: \"f9fc2de6-adf3-45c0-952a-e317f362aa51\") " pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:12.072144 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071826    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-os-release\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.072144 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071859    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-run-netns\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.072144 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071886    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-run-ovn\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.072144 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071911    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d169050-68f4-4fff-abb6-b261d3c5935b-ovnkube-config\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.072144 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071930    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d169050-68f4-4fff-abb6-b261d3c5935b-ovnkube-script-lib\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.072144 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.071953    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/486589ec-aa0a-497e-88e6-79e52df71588-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.072144 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072013    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1f8b093e-7ab9-406d-8a0e-c79e694cb205-kubelet-config\") pod \"global-pull-secret-syncer-5nws8\" (UID: \"1f8b093e-7ab9-406d-8a0e-c79e694cb205\") " pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:12.072144 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072048    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frsgq\" (UniqueName: \"kubernetes.io/projected/bd0f1bed-d043-4fbd-90d0-f3dfb22292de-kube-api-access-frsgq\") pod \"iptables-alerter-2pb8h\" (UID: \"bd0f1bed-d043-4fbd-90d0-f3dfb22292de\") " pod="openshift-network-operator/iptables-alerter-2pb8h"
Jul 17 18:33:12.072144 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072080    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-etc-kubernetes\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.072144 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072106    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpxrb\" (UniqueName: \"kubernetes.io/projected/3a1608b8-a53a-480d-965d-8ab7e3ec8f22-kube-api-access-mpxrb\") pod \"node-resolver-njrqr\" (UID: \"3a1608b8-a53a-480d-965d-8ab7e3ec8f22\") " pod="openshift-dns/node-resolver-njrqr"
Jul 17 18:33:12.072144 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072131    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1eb72ece-f355-4599-b236-299ac5dce970-sys-fs\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.072622 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072162    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-multus-cni-dir\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.072622 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072153    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-etc-kubernetes\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.072622 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072188    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-host-run-k8s-cni-cncf-io\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.072622 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072214    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-multus-conf-dir\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.072622 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072240    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8zht\" (UniqueName: \"kubernetes.io/projected/6d169050-68f4-4fff-abb6-b261d3c5935b-kube-api-access-l8zht\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.072622 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072255    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-host-run-k8s-cni-cncf-io\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.072622 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072262    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e35f12b9-1b2e-474f-8261-ec3e25a25775-konnectivity-ca\") pod \"konnectivity-agent-vtswf\" (UID: \"e35f12b9-1b2e-474f-8261-ec3e25a25775\") " pod="kube-system/konnectivity-agent-vtswf"
Jul 17 18:33:12.072622 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072298    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-multus-cni-dir\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.072622 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072331    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d33d089a-3e2f-4edd-a459-0303cf90f1af-serviceca\") pod \"node-ca-ztfrm\" (UID: \"d33d089a-3e2f-4edd-a459-0303cf90f1af\") " pod="openshift-image-registry/node-ca-ztfrm"
Jul 17 18:33:12.072622 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072357    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-etc-sysctl-d\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.072622 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072362    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-multus-conf-dir\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.072622 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072381    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1eb72ece-f355-4599-b236-299ac5dce970-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.072622 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072410    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-var-lib-openvswitch\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.072622 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072436    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/486589ec-aa0a-497e-88e6-79e52df71588-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.072622 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072461    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a1608b8-a53a-480d-965d-8ab7e3ec8f22-tmp-dir\") pod \"node-resolver-njrqr\" (UID: \"3a1608b8-a53a-480d-965d-8ab7e3ec8f22\") " pod="openshift-dns/node-resolver-njrqr"
Jul 17 18:33:12.072622 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072482    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d169050-68f4-4fff-abb6-b261d3c5935b-env-overrides\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.072622 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072484    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-etc-sysctl-d\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.073217 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072505    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-etc-kubernetes\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.073217 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072529    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-run-systemd\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.073217 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072555    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/486589ec-aa0a-497e-88e6-79e52df71588-system-cni-dir\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.073217 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072560    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-etc-kubernetes\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.073217 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072581    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-etc-sysctl-conf\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.073217 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072605    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1eb72ece-f355-4599-b236-299ac5dce970-registration-dir\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.073217 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072629    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-etc-openvswitch\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.073217 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072652    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-log-socket\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.073217 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072645    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e35f12b9-1b2e-474f-8261-ec3e25a25775-konnectivity-ca\") pod \"konnectivity-agent-vtswf\" (UID: \"e35f12b9-1b2e-474f-8261-ec3e25a25775\") " pod="kube-system/konnectivity-agent-vtswf"
Jul 17 18:33:12.073217 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072691    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-cni-netd\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.073217 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072727    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d169050-68f4-4fff-abb6-b261d3c5935b-ovn-node-metrics-cert\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.073217 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072755    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xljnf\" (UniqueName: \"kubernetes.io/projected/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-kube-api-access-xljnf\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.073217 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072786    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-etc-sysctl-conf\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.073217 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072800    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-host\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.073217 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072801    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d33d089a-3e2f-4edd-a459-0303cf90f1af-serviceca\") pod \"node-ca-ztfrm\" (UID: \"d33d089a-3e2f-4edd-a459-0303cf90f1af\") " pod="openshift-image-registry/node-ca-ztfrm"
Jul 17 18:33:12.073217 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072924    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrnc7\" (UniqueName: \"kubernetes.io/projected/1eb72ece-f355-4599-b236-299ac5dce970-kube-api-access-nrnc7\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.073217 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072946    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78304550-dac5-44f4-91e0-58503470a03a-host\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.074020 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.072993    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-systemd-units\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.074020 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.073027    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-run-ovn-kubernetes\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.074020 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.073076    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/486589ec-aa0a-497e-88e6-79e52df71588-os-release\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.074020 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.073105    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/486589ec-aa0a-497e-88e6-79e52df71588-cni-binary-copy\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.074020 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.073127    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e35f12b9-1b2e-474f-8261-ec3e25a25775-agent-certs\") pod \"konnectivity-agent-vtswf\" (UID: \"e35f12b9-1b2e-474f-8261-ec3e25a25775\") " pod="kube-system/konnectivity-agent-vtswf"
Jul 17 18:33:12.074020 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.073154    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1eb72ece-f355-4599-b236-299ac5dce970-socket-dir\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.074020 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.073168    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-run-openvswitch\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.074020 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.073187    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-node-log\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.074020 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.073203    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-host-run-netns\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.074020 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.073218    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs\") pod \"network-metrics-daemon-tt279\" (UID: \"f9fc2de6-adf3-45c0-952a-e317f362aa51\") " pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:12.074020 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.073240    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78304550-dac5-44f4-91e0-58503470a03a-tmp\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.074020 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.073254    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-cni-bin\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.074020 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.073267    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1f8b093e-7ab9-406d-8a0e-c79e694cb205-dbus\") pod \"global-pull-secret-syncer-5nws8\" (UID: \"1f8b093e-7ab9-406d-8a0e-c79e694cb205\") " pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:12.074020 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.073281    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd0f1bed-d043-4fbd-90d0-f3dfb22292de-host-slash\") pod \"iptables-alerter-2pb8h\" (UID: \"bd0f1bed-d043-4fbd-90d0-f3dfb22292de\") " pod="openshift-network-operator/iptables-alerter-2pb8h"
Jul 17 18:33:12.074020 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.073329    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-host-run-netns\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.074020 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.073365    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-hostroot\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.074020 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.073435    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd0f1bed-d043-4fbd-90d0-f3dfb22292de-host-slash\") pod \"iptables-alerter-2pb8h\" (UID: \"bd0f1bed-d043-4fbd-90d0-f3dfb22292de\") " pod="openshift-network-operator/iptables-alerter-2pb8h"
Jul 17 18:33:12.074869 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.073442    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-hostroot\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.074869 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:12.073525    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:12.074869 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:12.073703    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs podName:f9fc2de6-adf3-45c0-952a-e317f362aa51 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:12.573667943 +0000 UTC m=+3.015176639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs") pod "network-metrics-daemon-tt279" (UID: "f9fc2de6-adf3-45c0-952a-e317f362aa51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:12.076361 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.076337    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78304550-dac5-44f4-91e0-58503470a03a-tmp\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.076616 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.076595    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e35f12b9-1b2e-474f-8261-ec3e25a25775-agent-certs\") pod \"konnectivity-agent-vtswf\" (UID: \"e35f12b9-1b2e-474f-8261-ec3e25a25775\") " pod="kube-system/konnectivity-agent-vtswf"
Jul 17 18:33:12.076684 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.076642    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78304550-dac5-44f4-91e0-58503470a03a-etc-tuned\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.081899 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.081340    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lwz9s"
Jul 17 18:33:12.115411 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.115342    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58ndx\" (UniqueName: \"kubernetes.io/projected/f9fc2de6-adf3-45c0-952a-e317f362aa51-kube-api-access-58ndx\") pod \"network-metrics-daemon-tt279\" (UID: \"f9fc2de6-adf3-45c0-952a-e317f362aa51\") " pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:12.116446 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:12.116417    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:33:12.116446 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:12.116450    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:33:12.116597 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:12.116485    2559 projected.go:196] Error preparing data for projected volume kube-api-access-cpdgv for pod openshift-network-diagnostics/network-check-target-m6ftw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:12.116597 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:12.116557    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv podName:b6eaddd6-5198-4d18-bd28-bd1d31df6994 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:12.616536006 +0000 UTC m=+3.058044676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cpdgv" (UniqueName: "kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv") pod "network-check-target-m6ftw" (UID: "b6eaddd6-5198-4d18-bd28-bd1d31df6994") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:12.119066 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.119034    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frsgq\" (UniqueName: \"kubernetes.io/projected/bd0f1bed-d043-4fbd-90d0-f3dfb22292de-kube-api-access-frsgq\") pod \"iptables-alerter-2pb8h\" (UID: \"bd0f1bed-d043-4fbd-90d0-f3dfb22292de\") " pod="openshift-network-operator/iptables-alerter-2pb8h"
Jul 17 18:33:12.119776 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.119751    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hccv5\" (UniqueName: \"kubernetes.io/projected/d33d089a-3e2f-4edd-a459-0303cf90f1af-kube-api-access-hccv5\") pod \"node-ca-ztfrm\" (UID: \"d33d089a-3e2f-4edd-a459-0303cf90f1af\") " pod="openshift-image-registry/node-ca-ztfrm"
Jul 17 18:33:12.120699 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.120672    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xljnf\" (UniqueName: \"kubernetes.io/projected/57c3fb7d-dfc9-468c-a17a-37a625e1fb74-kube-api-access-xljnf\") pod \"multus-58rmc\" (UID: \"57c3fb7d-dfc9-468c-a17a-37a625e1fb74\") " pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.137835 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.137808    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dltsf\" (UniqueName: \"kubernetes.io/projected/78304550-dac5-44f4-91e0-58503470a03a-kube-api-access-dltsf\") pod \"tuned-nd8nj\" (UID: \"78304550-dac5-44f4-91e0-58503470a03a\") " pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.176362 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.173769    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1eb72ece-f355-4599-b236-299ac5dce970-socket-dir\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.176362 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.173815    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-run-openvswitch\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.176362 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.173847    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-node-log\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.176362 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.173898    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-cni-bin\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.176362 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.173924    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1f8b093e-7ab9-406d-8a0e-c79e694cb205-dbus\") pod \"global-pull-secret-syncer-5nws8\" (UID: \"1f8b093e-7ab9-406d-8a0e-c79e694cb205\") " pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:12.176362 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.173984    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.176362 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174013    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/486589ec-aa0a-497e-88e6-79e52df71588-cnibin\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.176362 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174042    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/486589ec-aa0a-497e-88e6-79e52df71588-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.176362 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174071    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a1608b8-a53a-480d-965d-8ab7e3ec8f22-hosts-file\") pod \"node-resolver-njrqr\" (UID: \"3a1608b8-a53a-480d-965d-8ab7e3ec8f22\") " pod="openshift-dns/node-resolver-njrqr"
Jul 17 18:33:12.176362 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174099    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1eb72ece-f355-4599-b236-299ac5dce970-device-dir\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.176362 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174127    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb7l2\" (UniqueName: \"kubernetes.io/projected/486589ec-aa0a-497e-88e6-79e52df71588-kube-api-access-bb7l2\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.176362 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174156    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret\") pod \"global-pull-secret-syncer-5nws8\" (UID: \"1f8b093e-7ab9-406d-8a0e-c79e694cb205\") " pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:12.176362 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174201    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-kubelet\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.176362 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174235    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1eb72ece-f355-4599-b236-299ac5dce970-etc-selinux\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.176362 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174264    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-slash\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.176362 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174296    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-run-netns\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.176362 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174339    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-run-ovn\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.177328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174367    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d169050-68f4-4fff-abb6-b261d3c5935b-ovnkube-config\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.177328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174395    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d169050-68f4-4fff-abb6-b261d3c5935b-ovnkube-script-lib\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.177328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174427    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/486589ec-aa0a-497e-88e6-79e52df71588-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.177328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174451    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1f8b093e-7ab9-406d-8a0e-c79e694cb205-kubelet-config\") pod \"global-pull-secret-syncer-5nws8\" (UID: \"1f8b093e-7ab9-406d-8a0e-c79e694cb205\") " pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:12.177328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174481    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpxrb\" (UniqueName: \"kubernetes.io/projected/3a1608b8-a53a-480d-965d-8ab7e3ec8f22-kube-api-access-mpxrb\") pod \"node-resolver-njrqr\" (UID: \"3a1608b8-a53a-480d-965d-8ab7e3ec8f22\") " pod="openshift-dns/node-resolver-njrqr"
Jul 17 18:33:12.177328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174511    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1eb72ece-f355-4599-b236-299ac5dce970-sys-fs\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.177328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174546    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8zht\" (UniqueName: \"kubernetes.io/projected/6d169050-68f4-4fff-abb6-b261d3c5935b-kube-api-access-l8zht\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.177328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174578    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1eb72ece-f355-4599-b236-299ac5dce970-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.177328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174602    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-var-lib-openvswitch\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.177328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174631    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/486589ec-aa0a-497e-88e6-79e52df71588-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.177328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174661    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a1608b8-a53a-480d-965d-8ab7e3ec8f22-tmp-dir\") pod \"node-resolver-njrqr\" (UID: \"3a1608b8-a53a-480d-965d-8ab7e3ec8f22\") " pod="openshift-dns/node-resolver-njrqr"
Jul 17 18:33:12.177328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174690    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d169050-68f4-4fff-abb6-b261d3c5935b-env-overrides\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.177328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174719    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-run-systemd\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.177328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174743    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/486589ec-aa0a-497e-88e6-79e52df71588-system-cni-dir\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.177328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174778    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1eb72ece-f355-4599-b236-299ac5dce970-registration-dir\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.177328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174813    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-etc-openvswitch\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.177328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174848    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-log-socket\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.178019 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174876    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-cni-netd\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.178019 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174904    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d169050-68f4-4fff-abb6-b261d3c5935b-ovn-node-metrics-cert\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.178019 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174931    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrnc7\" (UniqueName: \"kubernetes.io/projected/1eb72ece-f355-4599-b236-299ac5dce970-kube-api-access-nrnc7\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.178019 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174959    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-systemd-units\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.178019 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.174988    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-run-ovn-kubernetes\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.178019 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.175017    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/486589ec-aa0a-497e-88e6-79e52df71588-os-release\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.178019 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.175050    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/486589ec-aa0a-497e-88e6-79e52df71588-cni-binary-copy\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.178019 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.175694    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/486589ec-aa0a-497e-88e6-79e52df71588-cni-binary-copy\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.178019 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.175826    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1eb72ece-f355-4599-b236-299ac5dce970-socket-dir\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.178019 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.175884    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-run-openvswitch\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.178019 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.175930    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-node-log\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.178019 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.175974    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-cni-bin\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.178019 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.176039    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1f8b093e-7ab9-406d-8a0e-c79e694cb205-dbus\") pod \"global-pull-secret-syncer-5nws8\" (UID: \"1f8b093e-7ab9-406d-8a0e-c79e694cb205\") " pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:12.178019 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.176097    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-cni-netd\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.178019 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.176100    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.178019 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.176103    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1eb72ece-f355-4599-b236-299ac5dce970-registration-dir\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.178019 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.176215    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-kubelet\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.178709 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.176229    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1eb72ece-f355-4599-b236-299ac5dce970-device-dir\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.178709 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.176276    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a1608b8-a53a-480d-965d-8ab7e3ec8f22-hosts-file\") pod \"node-resolver-njrqr\" (UID: \"3a1608b8-a53a-480d-965d-8ab7e3ec8f22\") " pod="openshift-dns/node-resolver-njrqr"
Jul 17 18:33:12.178709 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.176333    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/486589ec-aa0a-497e-88e6-79e52df71588-cnibin\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.178709 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.176408    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-run-systemd\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.178881 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.176433    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-systemd-units\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.178881 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.176457    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-log-socket\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.178881 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.176500    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/486589ec-aa0a-497e-88e6-79e52df71588-os-release\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.178881 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.176524    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-run-ovn-kubernetes\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.178881 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.176738    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/486589ec-aa0a-497e-88e6-79e52df71588-system-cni-dir\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.178881 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.176810    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-var-lib-openvswitch\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.178881 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.176822    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-run-netns\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.179192 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.176862    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1eb72ece-f355-4599-b236-299ac5dce970-sys-fs\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.179192 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.176890    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-host-slash\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.179192 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:12.177008    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:12.179192 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:12.178975    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret podName:1f8b093e-7ab9-406d-8a0e-c79e694cb205 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:12.678951907 +0000 UTC m=+3.120460584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret") pod "global-pull-secret-syncer-5nws8" (UID: "1f8b093e-7ab9-406d-8a0e-c79e694cb205") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:12.179192 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.177214    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-etc-openvswitch\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.179192 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.177230    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/486589ec-aa0a-497e-88e6-79e52df71588-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.179192 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.177371    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1eb72ece-f355-4599-b236-299ac5dce970-etc-selinux\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.179192 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.177413    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d169050-68f4-4fff-abb6-b261d3c5935b-run-ovn\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.179192 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.177590    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/486589ec-aa0a-497e-88e6-79e52df71588-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.179192 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.177630    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d169050-68f4-4fff-abb6-b261d3c5935b-env-overrides\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.179192 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.177639    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a1608b8-a53a-480d-965d-8ab7e3ec8f22-tmp-dir\") pod \"node-resolver-njrqr\" (UID: \"3a1608b8-a53a-480d-965d-8ab7e3ec8f22\") " pod="openshift-dns/node-resolver-njrqr"
Jul 17 18:33:12.179192 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.177713    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1eb72ece-f355-4599-b236-299ac5dce970-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.179192 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.177714    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d169050-68f4-4fff-abb6-b261d3c5935b-ovnkube-script-lib\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.179192 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.177865    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d169050-68f4-4fff-abb6-b261d3c5935b-ovnkube-config\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.179192 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.177898    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/486589ec-aa0a-497e-88e6-79e52df71588-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.179192 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.177167    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1f8b093e-7ab9-406d-8a0e-c79e694cb205-kubelet-config\") pod \"global-pull-secret-syncer-5nws8\" (UID: \"1f8b093e-7ab9-406d-8a0e-c79e694cb205\") " pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:12.179812 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.179541    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d169050-68f4-4fff-abb6-b261d3c5935b-ovn-node-metrics-cert\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.222814 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.222745    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpxrb\" (UniqueName: \"kubernetes.io/projected/3a1608b8-a53a-480d-965d-8ab7e3ec8f22-kube-api-access-mpxrb\") pod \"node-resolver-njrqr\" (UID: \"3a1608b8-a53a-480d-965d-8ab7e3ec8f22\") " pod="openshift-dns/node-resolver-njrqr"
Jul 17 18:33:12.222814 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.222783    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb7l2\" (UniqueName: \"kubernetes.io/projected/486589ec-aa0a-497e-88e6-79e52df71588-kube-api-access-bb7l2\") pod \"multus-additional-cni-plugins-cnj2q\" (UID: \"486589ec-aa0a-497e-88e6-79e52df71588\") " pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.223146 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.223129    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8zht\" (UniqueName: \"kubernetes.io/projected/6d169050-68f4-4fff-abb6-b261d3c5935b-kube-api-access-l8zht\") pod \"ovnkube-node-xhn5c\" (UID: \"6d169050-68f4-4fff-abb6-b261d3c5935b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.224403 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.224335    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrnc7\" (UniqueName: \"kubernetes.io/projected/1eb72ece-f355-4599-b236-299ac5dce970-kube-api-access-nrnc7\") pod \"aws-ebs-csi-driver-node-fvgdz\" (UID: \"1eb72ece-f355-4599-b236-299ac5dce970\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.265551 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.265512    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2pb8h"
Jul 17 18:33:12.276098 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.276075    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vtswf"
Jul 17 18:33:12.284048 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.284028    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ztfrm"
Jul 17 18:33:12.290810 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.290789    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-58rmc"
Jul 17 18:33:12.297860 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.297833    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nd8nj"
Jul 17 18:33:12.312386 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.312368    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cnj2q"
Jul 17 18:33:12.319231 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.319211    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-njrqr"
Jul 17 18:33:12.325113 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.325096    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz"
Jul 17 18:33:12.330776 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.330758    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:12.577696 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.577600    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs\") pod \"network-metrics-daemon-tt279\" (UID: \"f9fc2de6-adf3-45c0-952a-e317f362aa51\") " pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:12.577853 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:12.577725    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:12.577853 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:12.577803    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs podName:f9fc2de6-adf3-45c0-952a-e317f362aa51 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:13.577781444 +0000 UTC m=+4.019290111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs") pod "network-metrics-daemon-tt279" (UID: "f9fc2de6-adf3-45c0-952a-e317f362aa51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:12.678412 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.678368    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpdgv\" (UniqueName: \"kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv\") pod \"network-check-target-m6ftw\" (UID: \"b6eaddd6-5198-4d18-bd28-bd1d31df6994\") " pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:12.678563 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:12.678539    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:33:12.678563 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:12.678560    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:33:12.678660 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:12.678570    2559 projected.go:196] Error preparing data for projected volume kube-api-access-cpdgv for pod openshift-network-diagnostics/network-check-target-m6ftw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:12.678660 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:12.678632    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv podName:b6eaddd6-5198-4d18-bd28-bd1d31df6994 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:13.678618735 +0000 UTC m=+4.120127406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cpdgv" (UniqueName: "kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv") pod "network-check-target-m6ftw" (UID: "b6eaddd6-5198-4d18-bd28-bd1d31df6994") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:12.779106 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:12.779077    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret\") pod \"global-pull-secret-syncer-5nws8\" (UID: \"1f8b093e-7ab9-406d-8a0e-c79e694cb205\") " pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:12.779238 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:12.779218    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:12.779295 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:12.779277    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret podName:1f8b093e-7ab9-406d-8a0e-c79e694cb205 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:13.779263805 +0000 UTC m=+4.220772488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret") pod "global-pull-secret-syncer-5nws8" (UID: "1f8b093e-7ab9-406d-8a0e-c79e694cb205") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:13.084948 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:13.084898    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 18:28:12 +0000 UTC" deadline="2028-03-28 17:15:08.373740826 +0000 UTC"
Jul 17 18:33:13.084948 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:13.084939    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14878h41m55.288805987s"
Jul 17 18:33:13.110584 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:13.110475    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c3fb7d_dfc9_468c_a17a_37a625e1fb74.slice/crio-58026a0fe93365ac4e258f17895368f2d3231d65e56d74d8c7df550dfa2f6c36 WatchSource:0}: Error finding container 58026a0fe93365ac4e258f17895368f2d3231d65e56d74d8c7df550dfa2f6c36: Status 404 returned error can't find the container with id 58026a0fe93365ac4e258f17895368f2d3231d65e56d74d8c7df550dfa2f6c36
Jul 17 18:33:13.112039 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:13.111957    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d169050_68f4_4fff_abb6_b261d3c5935b.slice/crio-784ca7d4e53149dca77a9b4ea10315a59ac9bbd3407fdd864e65edae79bc50fc WatchSource:0}: Error finding container 784ca7d4e53149dca77a9b4ea10315a59ac9bbd3407fdd864e65edae79bc50fc: Status 404 returned error can't find the container with id 784ca7d4e53149dca77a9b4ea10315a59ac9bbd3407fdd864e65edae79bc50fc
Jul 17 18:33:13.114776 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:13.114215    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486589ec_aa0a_497e_88e6_79e52df71588.slice/crio-c5b129f7d7b6301d4eff1e7e4dc86194e691cc9e417ecc93d6789b7013f0ebef WatchSource:0}: Error finding container c5b129f7d7b6301d4eff1e7e4dc86194e691cc9e417ecc93d6789b7013f0ebef: Status 404 returned error can't find the container with id c5b129f7d7b6301d4eff1e7e4dc86194e691cc9e417ecc93d6789b7013f0ebef
Jul 17 18:33:13.114776 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:13.114619    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd33d089a_3e2f_4edd_a459_0303cf90f1af.slice/crio-676c5f869d3d99d486d5d82799102ab2d357dca5cb00cd5613f42a7cb1c63bdf WatchSource:0}: Error finding container 676c5f869d3d99d486d5d82799102ab2d357dca5cb00cd5613f42a7cb1c63bdf: Status 404 returned error can't find the container with id 676c5f869d3d99d486d5d82799102ab2d357dca5cb00cd5613f42a7cb1c63bdf
Jul 17 18:33:13.118377 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:13.118354    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1608b8_a53a_480d_965d_8ab7e3ec8f22.slice/crio-4986db97f09b23c1ff068d6f346aa4aaf96e3f660c0b5acdd690cfbca245723e WatchSource:0}: Error finding container 4986db97f09b23c1ff068d6f346aa4aaf96e3f660c0b5acdd690cfbca245723e: Status 404 returned error can't find the container with id 4986db97f09b23c1ff068d6f346aa4aaf96e3f660c0b5acdd690cfbca245723e
Jul 17 18:33:13.118883 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:13.118866    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78304550_dac5_44f4_91e0_58503470a03a.slice/crio-9e560de1713135aedf98ee7a1a602a5251f56278ee34356401ed9cea1709db2a WatchSource:0}: Error finding container 9e560de1713135aedf98ee7a1a602a5251f56278ee34356401ed9cea1709db2a: Status 404 returned error can't find the container with id 9e560de1713135aedf98ee7a1a602a5251f56278ee34356401ed9cea1709db2a
Jul 17 18:33:13.120616 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:13.120594    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd0f1bed_d043_4fbd_90d0_f3dfb22292de.slice/crio-88ca75e72e97f5dd79f6490dac0b792987c079a76b206ed6999e3c8be4898361 WatchSource:0}: Error finding container 88ca75e72e97f5dd79f6490dac0b792987c079a76b206ed6999e3c8be4898361: Status 404 returned error can't find the container with id 88ca75e72e97f5dd79f6490dac0b792987c079a76b206ed6999e3c8be4898361
Jul 17 18:33:13.121364 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:13.121343    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35f12b9_1b2e_474f_8261_ec3e25a25775.slice/crio-20eca71274489f9ca64c634ddee98a8f83f86487d9ace16f15924729be2025bd WatchSource:0}: Error finding container 20eca71274489f9ca64c634ddee98a8f83f86487d9ace16f15924729be2025bd: Status 404 returned error can't find the container with id 20eca71274489f9ca64c634ddee98a8f83f86487d9ace16f15924729be2025bd
Jul 17 18:33:13.123283 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:13.123265    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb72ece_f355_4599_b236_299ac5dce970.slice/crio-c55dd36870ebabe0d733de46b0c5c8db381b6ed41898be82db51d43c84919989 WatchSource:0}: Error finding container c55dd36870ebabe0d733de46b0c5c8db381b6ed41898be82db51d43c84919989: Status 404 returned error can't find the container with id c55dd36870ebabe0d733de46b0c5c8db381b6ed41898be82db51d43c84919989
Jul 17 18:33:13.130698 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:13.130667    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cnj2q" event={"ID":"486589ec-aa0a-497e-88e6-79e52df71588","Type":"ContainerStarted","Data":"c5b129f7d7b6301d4eff1e7e4dc86194e691cc9e417ecc93d6789b7013f0ebef"}
Jul 17 18:33:13.131977 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:13.131948    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nd8nj" event={"ID":"78304550-dac5-44f4-91e0-58503470a03a","Type":"ContainerStarted","Data":"9e560de1713135aedf98ee7a1a602a5251f56278ee34356401ed9cea1709db2a"}
Jul 17 18:33:13.133749 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:13.133717    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz" event={"ID":"1eb72ece-f355-4599-b236-299ac5dce970","Type":"ContainerStarted","Data":"c55dd36870ebabe0d733de46b0c5c8db381b6ed41898be82db51d43c84919989"}
Jul 17 18:33:13.134868 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:13.134842    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-njrqr" event={"ID":"3a1608b8-a53a-480d-965d-8ab7e3ec8f22","Type":"ContainerStarted","Data":"4986db97f09b23c1ff068d6f346aa4aaf96e3f660c0b5acdd690cfbca245723e"}
Jul 17 18:33:13.136131 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:13.136109    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ztfrm" event={"ID":"d33d089a-3e2f-4edd-a459-0303cf90f1af","Type":"ContainerStarted","Data":"676c5f869d3d99d486d5d82799102ab2d357dca5cb00cd5613f42a7cb1c63bdf"}
Jul 17 18:33:13.137466 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:13.137444    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c" event={"ID":"6d169050-68f4-4fff-abb6-b261d3c5935b","Type":"ContainerStarted","Data":"784ca7d4e53149dca77a9b4ea10315a59ac9bbd3407fdd864e65edae79bc50fc"}
Jul 17 18:33:13.138679 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:13.138660    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-58rmc" event={"ID":"57c3fb7d-dfc9-468c-a17a-37a625e1fb74","Type":"ContainerStarted","Data":"58026a0fe93365ac4e258f17895368f2d3231d65e56d74d8c7df550dfa2f6c36"}
Jul 17 18:33:13.139876 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:13.139850    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vtswf" event={"ID":"e35f12b9-1b2e-474f-8261-ec3e25a25775","Type":"ContainerStarted","Data":"20eca71274489f9ca64c634ddee98a8f83f86487d9ace16f15924729be2025bd"}
Jul 17 18:33:13.140971 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:13.140953    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2pb8h" event={"ID":"bd0f1bed-d043-4fbd-90d0-f3dfb22292de","Type":"ContainerStarted","Data":"88ca75e72e97f5dd79f6490dac0b792987c079a76b206ed6999e3c8be4898361"}
Jul 17 18:33:13.584052 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:13.583753    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs\") pod \"network-metrics-daemon-tt279\" (UID: \"f9fc2de6-adf3-45c0-952a-e317f362aa51\") " pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:13.584249 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:13.584132    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:13.584249 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:13.584200    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs podName:f9fc2de6-adf3-45c0-952a-e317f362aa51 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:15.584179901 +0000 UTC m=+6.025688571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs") pod "network-metrics-daemon-tt279" (UID: "f9fc2de6-adf3-45c0-952a-e317f362aa51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:13.684581 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:13.684542    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpdgv\" (UniqueName: \"kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv\") pod \"network-check-target-m6ftw\" (UID: \"b6eaddd6-5198-4d18-bd28-bd1d31df6994\") " pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:13.684745 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:13.684714    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:33:13.684745 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:13.684733    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:33:13.684745 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:13.684744    2559 projected.go:196] Error preparing data for projected volume kube-api-access-cpdgv for pod openshift-network-diagnostics/network-check-target-m6ftw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:13.684901 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:13.684801    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv podName:b6eaddd6-5198-4d18-bd28-bd1d31df6994 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:15.68478199 +0000 UTC m=+6.126290674 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cpdgv" (UniqueName: "kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv") pod "network-check-target-m6ftw" (UID: "b6eaddd6-5198-4d18-bd28-bd1d31df6994") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:13.786140 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:13.785569    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret\") pod \"global-pull-secret-syncer-5nws8\" (UID: \"1f8b093e-7ab9-406d-8a0e-c79e694cb205\") " pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:13.786140 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:13.785729    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:13.786140 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:13.785789    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret podName:1f8b093e-7ab9-406d-8a0e-c79e694cb205 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:15.785770988 +0000 UTC m=+6.227279661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret") pod "global-pull-secret-syncer-5nws8" (UID: "1f8b093e-7ab9-406d-8a0e-c79e694cb205") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:14.087390 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:14.087291    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 18:28:12 +0000 UTC" deadline="2028-01-29 12:49:44.953798121 +0000 UTC"
Jul 17 18:33:14.087390 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:14.087389    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13458h16m30.866413752s"
Jul 17 18:33:14.120468 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:14.120435    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:14.120627 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:14.120582    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt279" podUID="f9fc2de6-adf3-45c0-952a-e317f362aa51"
Jul 17 18:33:14.121022 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:14.121002    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:14.121157 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:14.121101    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nws8" podUID="1f8b093e-7ab9-406d-8a0e-c79e694cb205"
Jul 17 18:33:14.121382 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:14.121363    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:14.121475 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:14.121456    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6ftw" podUID="b6eaddd6-5198-4d18-bd28-bd1d31df6994"
Jul 17 18:33:14.155069 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:14.154445    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-51.ec2.internal" event={"ID":"68a4b3cbdf87f36c033c410a5f90d655","Type":"ContainerStarted","Data":"b5203ae83c3cf3547c93a9509ec34a889a0a5d3364b4cd34c3a8620dcceca16c"}
Jul 17 18:33:14.161105 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:14.160574    2559 generic.go:356] "Generic (PLEG): container finished" podID="80efdace1bee045fa6ee7c757931d822" containerID="1c7026d6d92799fb0432e714de47e3197a3d5b711f934f84f9f77f68493bae44" exitCode=0
Jul 17 18:33:14.161105 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:14.160617    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal" event={"ID":"80efdace1bee045fa6ee7c757931d822","Type":"ContainerDied","Data":"1c7026d6d92799fb0432e714de47e3197a3d5b711f934f84f9f77f68493bae44"}
Jul 17 18:33:14.221587 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:14.221523    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-51.ec2.internal" podStartSLOduration=3.221498381 podStartE2EDuration="3.221498381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:14.186043356 +0000 UTC m=+4.627552048" watchObservedRunningTime="2026-07-17 18:33:14.221498381 +0000 UTC m=+4.663007058"
Jul 17 18:33:15.173935 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:15.173899    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal" event={"ID":"80efdace1bee045fa6ee7c757931d822","Type":"ContainerStarted","Data":"86839774ca8ad2210150b3d3e656b50c63f5b3754cd7721975654294220ef85d"}
Jul 17 18:33:15.599897 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:15.599813    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs\") pod \"network-metrics-daemon-tt279\" (UID: \"f9fc2de6-adf3-45c0-952a-e317f362aa51\") " pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:15.600048 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:15.600019    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:15.600114 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:15.600099    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs podName:f9fc2de6-adf3-45c0-952a-e317f362aa51 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:19.600079457 +0000 UTC m=+10.041588141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs") pod "network-metrics-daemon-tt279" (UID: "f9fc2de6-adf3-45c0-952a-e317f362aa51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:15.700836 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:15.700797    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpdgv\" (UniqueName: \"kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv\") pod \"network-check-target-m6ftw\" (UID: \"b6eaddd6-5198-4d18-bd28-bd1d31df6994\") " pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:15.701017 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:15.700974    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:33:15.701017 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:15.700994    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:33:15.701017 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:15.701005    2559 projected.go:196] Error preparing data for projected volume kube-api-access-cpdgv for pod openshift-network-diagnostics/network-check-target-m6ftw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:15.701167 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:15.701061    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv podName:b6eaddd6-5198-4d18-bd28-bd1d31df6994 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:19.701042559 +0000 UTC m=+10.142551245 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cpdgv" (UniqueName: "kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv") pod "network-check-target-m6ftw" (UID: "b6eaddd6-5198-4d18-bd28-bd1d31df6994") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:15.801863 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:15.801823    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret\") pod \"global-pull-secret-syncer-5nws8\" (UID: \"1f8b093e-7ab9-406d-8a0e-c79e694cb205\") " pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:15.802038 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:15.802021    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:15.802114 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:15.802088    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret podName:1f8b093e-7ab9-406d-8a0e-c79e694cb205 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:19.802069742 +0000 UTC m=+10.243578411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret") pod "global-pull-secret-syncer-5nws8" (UID: "1f8b093e-7ab9-406d-8a0e-c79e694cb205") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:16.119840 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:16.119795    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:16.120028 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:16.119954    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6ftw" podUID="b6eaddd6-5198-4d18-bd28-bd1d31df6994"
Jul 17 18:33:16.120281 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:16.120257    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:16.120407 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:16.120391    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt279" podUID="f9fc2de6-adf3-45c0-952a-e317f362aa51"
Jul 17 18:33:16.120563 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:16.120539    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:16.120703 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:16.120652    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nws8" podUID="1f8b093e-7ab9-406d-8a0e-c79e694cb205"
Jul 17 18:33:18.120526 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:18.120449    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:18.120526 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:18.120504    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:18.121009 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:18.120578    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt279" podUID="f9fc2de6-adf3-45c0-952a-e317f362aa51"
Jul 17 18:33:18.121009 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:18.120652    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:18.121009 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:18.120748    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6ftw" podUID="b6eaddd6-5198-4d18-bd28-bd1d31df6994"
Jul 17 18:33:18.121009 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:18.120831    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nws8" podUID="1f8b093e-7ab9-406d-8a0e-c79e694cb205"
Jul 17 18:33:19.635081 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:19.635042    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs\") pod \"network-metrics-daemon-tt279\" (UID: \"f9fc2de6-adf3-45c0-952a-e317f362aa51\") " pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:19.635519 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:19.635190    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:19.635519 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:19.635260    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs podName:f9fc2de6-adf3-45c0-952a-e317f362aa51 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:27.635239932 +0000 UTC m=+18.076748602 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs") pod "network-metrics-daemon-tt279" (UID: "f9fc2de6-adf3-45c0-952a-e317f362aa51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:19.736028 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:19.735996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpdgv\" (UniqueName: \"kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv\") pod \"network-check-target-m6ftw\" (UID: \"b6eaddd6-5198-4d18-bd28-bd1d31df6994\") " pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:19.736202 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:19.736138    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:33:19.736202 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:19.736152    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:33:19.736202 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:19.736161    2559 projected.go:196] Error preparing data for projected volume kube-api-access-cpdgv for pod openshift-network-diagnostics/network-check-target-m6ftw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:19.736353 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:19.736211    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv podName:b6eaddd6-5198-4d18-bd28-bd1d31df6994 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:27.736192924 +0000 UTC m=+18.177701609 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cpdgv" (UniqueName: "kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv") pod "network-check-target-m6ftw" (UID: "b6eaddd6-5198-4d18-bd28-bd1d31df6994") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:19.836528 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:19.836491    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret\") pod \"global-pull-secret-syncer-5nws8\" (UID: \"1f8b093e-7ab9-406d-8a0e-c79e694cb205\") " pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:19.836719 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:19.836616    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:19.836719 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:19.836688    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret podName:1f8b093e-7ab9-406d-8a0e-c79e694cb205 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:27.836668294 +0000 UTC m=+18.278176965 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret") pod "global-pull-secret-syncer-5nws8" (UID: "1f8b093e-7ab9-406d-8a0e-c79e694cb205") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:20.120280 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:20.120205    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:20.120439 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:20.120297    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nws8" podUID="1f8b093e-7ab9-406d-8a0e-c79e694cb205"
Jul 17 18:33:20.120651 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:20.120632    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:20.120751 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:20.120719    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6ftw" podUID="b6eaddd6-5198-4d18-bd28-bd1d31df6994"
Jul 17 18:33:20.120812 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:20.120787    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:20.120864 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:20.120848    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt279" podUID="f9fc2de6-adf3-45c0-952a-e317f362aa51"
Jul 17 18:33:22.121661 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:22.121623    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:22.122053 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:22.121755    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nws8" podUID="1f8b093e-7ab9-406d-8a0e-c79e694cb205"
Jul 17 18:33:22.122110 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:22.122072    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:22.122142 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:22.122130    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6ftw" podUID="b6eaddd6-5198-4d18-bd28-bd1d31df6994"
Jul 17 18:33:22.122186 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:22.122171    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:22.122241 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:22.122217    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt279" podUID="f9fc2de6-adf3-45c0-952a-e317f362aa51"
Jul 17 18:33:24.119497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:24.119463    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:24.119497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:24.119494    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:24.119967 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:24.119463    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:24.119967 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:24.119588    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nws8" podUID="1f8b093e-7ab9-406d-8a0e-c79e694cb205"
Jul 17 18:33:24.119967 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:24.119668    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6ftw" podUID="b6eaddd6-5198-4d18-bd28-bd1d31df6994"
Jul 17 18:33:24.119967 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:24.119777    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt279" podUID="f9fc2de6-adf3-45c0-952a-e317f362aa51"
Jul 17 18:33:26.119396 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:26.119360    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:26.119823 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:26.119357    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:26.119823 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:26.119488    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:26.119823 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:26.119490    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt279" podUID="f9fc2de6-adf3-45c0-952a-e317f362aa51"
Jul 17 18:33:26.119823 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:26.119553    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6ftw" podUID="b6eaddd6-5198-4d18-bd28-bd1d31df6994"
Jul 17 18:33:26.119823 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:26.119676    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nws8" podUID="1f8b093e-7ab9-406d-8a0e-c79e694cb205"
Jul 17 18:33:27.693422 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:27.693381    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs\") pod \"network-metrics-daemon-tt279\" (UID: \"f9fc2de6-adf3-45c0-952a-e317f362aa51\") " pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:27.693823 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:27.693547    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:27.693823 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:27.693613    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs podName:f9fc2de6-adf3-45c0-952a-e317f362aa51 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:43.693594225 +0000 UTC m=+34.135102894 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs") pod "network-metrics-daemon-tt279" (UID: "f9fc2de6-adf3-45c0-952a-e317f362aa51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:27.794112 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:27.794077    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpdgv\" (UniqueName: \"kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv\") pod \"network-check-target-m6ftw\" (UID: \"b6eaddd6-5198-4d18-bd28-bd1d31df6994\") " pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:27.794281 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:27.794233    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:33:27.794281 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:27.794251    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:33:27.794281 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:27.794262    2559 projected.go:196] Error preparing data for projected volume kube-api-access-cpdgv for pod openshift-network-diagnostics/network-check-target-m6ftw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:27.794429 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:27.794336    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv podName:b6eaddd6-5198-4d18-bd28-bd1d31df6994 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:43.794317303 +0000 UTC m=+34.235825989 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cpdgv" (UniqueName: "kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv") pod "network-check-target-m6ftw" (UID: "b6eaddd6-5198-4d18-bd28-bd1d31df6994") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:27.895471 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:27.895383    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret\") pod \"global-pull-secret-syncer-5nws8\" (UID: \"1f8b093e-7ab9-406d-8a0e-c79e694cb205\") " pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:27.895643 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:27.895569    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:27.895695 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:27.895656    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret podName:1f8b093e-7ab9-406d-8a0e-c79e694cb205 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:43.895633991 +0000 UTC m=+34.337142662 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret") pod "global-pull-secret-syncer-5nws8" (UID: "1f8b093e-7ab9-406d-8a0e-c79e694cb205") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:28.119658 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:28.119531    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:28.119658 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:28.119570    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:28.119658 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:28.119534    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:28.119879 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:28.119676    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt279" podUID="f9fc2de6-adf3-45c0-952a-e317f362aa51"
Jul 17 18:33:28.119879 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:28.119765    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nws8" podUID="1f8b093e-7ab9-406d-8a0e-c79e694cb205"
Jul 17 18:33:28.119879 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:28.119826    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6ftw" podUID="b6eaddd6-5198-4d18-bd28-bd1d31df6994"
Jul 17 18:33:30.120558 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:30.120525    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:30.120983 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:30.120634    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6ftw" podUID="b6eaddd6-5198-4d18-bd28-bd1d31df6994"
Jul 17 18:33:30.121085 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:30.121069    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:30.121195 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:30.121177    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt279" podUID="f9fc2de6-adf3-45c0-952a-e317f362aa51"
Jul 17 18:33:30.121282 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:30.121254    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:30.121400 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:30.121374    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nws8" podUID="1f8b093e-7ab9-406d-8a0e-c79e694cb205"
Jul 17 18:33:32.119478 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:32.119202    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:32.120196 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:32.119206    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:32.120196 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:32.119580    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nws8" podUID="1f8b093e-7ab9-406d-8a0e-c79e694cb205"
Jul 17 18:33:32.120196 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:32.119202    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:32.120196 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:32.119682    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt279" podUID="f9fc2de6-adf3-45c0-952a-e317f362aa51"
Jul 17 18:33:32.120196 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:32.119786    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6ftw" podUID="b6eaddd6-5198-4d18-bd28-bd1d31df6994"
Jul 17 18:33:32.208377 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:32.208344    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nd8nj" event={"ID":"78304550-dac5-44f4-91e0-58503470a03a","Type":"ContainerStarted","Data":"9245e192dbc0f5935eb5cf94b98d3c2a8de06a91baa051b9001ac271dcf72db9"}
Jul 17 18:33:32.210534 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:32.210503    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz" event={"ID":"1eb72ece-f355-4599-b236-299ac5dce970","Type":"ContainerStarted","Data":"f7e8fff88fc16f99d3ef548d191061b1d998c370bcb215a70d955821bd616a3c"}
Jul 17 18:33:32.212973 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:32.212948    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-njrqr" event={"ID":"3a1608b8-a53a-480d-965d-8ab7e3ec8f22","Type":"ContainerStarted","Data":"5e2542941b9de37b0b87602a0f6a4ead4ebdd2a48d004d8d2d65427446f9cfbb"}
Jul 17 18:33:32.214470 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:32.214434    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ztfrm" event={"ID":"d33d089a-3e2f-4edd-a459-0303cf90f1af","Type":"ContainerStarted","Data":"5d83eb34fc5d021e50e2d07e35c961e766dc24471c1bdf558f0d1b4e46cadbfd"}
Jul 17 18:33:32.215913 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:32.215897    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 18:33:32.216295 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:32.216269    2559 generic.go:356] "Generic (PLEG): container finished" podID="6d169050-68f4-4fff-abb6-b261d3c5935b" containerID="b426b6ed87955b298a263a25cfa93888ee65c62a061a7d282e43a762dcda0aee" exitCode=1
Jul 17 18:33:32.216367 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:32.216332    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c" event={"ID":"6d169050-68f4-4fff-abb6-b261d3c5935b","Type":"ContainerDied","Data":"b426b6ed87955b298a263a25cfa93888ee65c62a061a7d282e43a762dcda0aee"}
Jul 17 18:33:32.216411 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:32.216396    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c" event={"ID":"6d169050-68f4-4fff-abb6-b261d3c5935b","Type":"ContainerStarted","Data":"e6a1c84cdc474d2d85f8d8d52a829392b631288b6c1baf8f3b15f6a3445c7cda"}
Jul 17 18:33:32.218022 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:32.217988    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-58rmc" event={"ID":"57c3fb7d-dfc9-468c-a17a-37a625e1fb74","Type":"ContainerStarted","Data":"9abd582d5f3b73b1e2597b958da65c80ee086d2e3b4dac1b745d2dd13c3d0763"}
Jul 17 18:33:32.219674 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:32.219648    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vtswf" event={"ID":"e35f12b9-1b2e-474f-8261-ec3e25a25775","Type":"ContainerStarted","Data":"c4f28b242f101730d7772c259e4dd6ae470758c8af0e33435223b1c9a471b3ce"}
Jul 17 18:33:32.221181 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:32.221154    2559 generic.go:356] "Generic (PLEG): container finished" podID="486589ec-aa0a-497e-88e6-79e52df71588" containerID="36cff2d466b5e6744e19455bb7ddabfa305a538d58936237b477c1e404f7d9e8" exitCode=0
Jul 17 18:33:32.221276 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:32.221195    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cnj2q" event={"ID":"486589ec-aa0a-497e-88e6-79e52df71588","Type":"ContainerDied","Data":"36cff2d466b5e6744e19455bb7ddabfa305a538d58936237b477c1e404f7d9e8"}
Jul 17 18:33:32.231654 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:32.231617    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nd8nj" podStartSLOduration=3.546702323 podStartE2EDuration="22.231601896s" totalImagesPullingTime="18.684899573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:32.231455661 +0000 UTC m=+22.672964349" watchObservedRunningTime="2026-07-17 18:33:32.231601896 +0000 UTC m=+22.673110584"
Jul 17 18:33:32.231776 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:32.231742    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-51.ec2.internal" podStartSLOduration=21.23173049 podStartE2EDuration="21.23173049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:15.205296943 +0000 UTC m=+5.646805633" watchObservedRunningTime="2026-07-17 18:33:32.23173049 +0000 UTC m=+22.673239196"
Jul 17 18:33:32.252372 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:32.252331    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ztfrm" podStartSLOduration=3.586586019 podStartE2EDuration="22.252295476s" totalImagesPullingTime="18.665709457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:32.252176164 +0000 UTC m=+22.693684852" watchObservedRunningTime="2026-07-17 18:33:32.252295476 +0000 UTC m=+22.693804163"
Jul 17 18:33:32.277417 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:32.277366    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-58rmc" podStartSLOduration=3.5415903220000002 podStartE2EDuration="22.277351413s" totalImagesPullingTime="18.735761091s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:32.277270836 +0000 UTC m=+22.718779524" watchObservedRunningTime="2026-07-17 18:33:32.277351413 +0000 UTC m=+22.718860101"
Jul 17 18:33:32.334935 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:32.334885    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-njrqr" podStartSLOduration=3.780314943 podStartE2EDuration="22.334864064s" totalImagesPullingTime="18.554549121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:32.334784402 +0000 UTC m=+22.776293091" watchObservedRunningTime="2026-07-17 18:33:32.334864064 +0000 UTC m=+22.776372754"
Jul 17 18:33:33.235449 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:33.235159    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 18:33:33.236087 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:33.235910    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c" event={"ID":"6d169050-68f4-4fff-abb6-b261d3c5935b","Type":"ContainerStarted","Data":"303bc6d2cd9d5a0e3abc2fb186701c9e4b1115d58e4853df75c7ee9cc128c6fa"}
Jul 17 18:33:33.236087 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:33.236047    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c" event={"ID":"6d169050-68f4-4fff-abb6-b261d3c5935b","Type":"ContainerStarted","Data":"55f0ea11eb591e6183d32742e63e0e54f6fed4ba2e1841dfcbce31adfe265384"}
Jul 17 18:33:33.236207 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:33.236104    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c" event={"ID":"6d169050-68f4-4fff-abb6-b261d3c5935b","Type":"ContainerStarted","Data":"0b55911852af8e5e9a8760a3df2a1f6360735cf9c720f6c43dd11807608e7807"}
Jul 17 18:33:33.236207 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:33.236155    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c" event={"ID":"6d169050-68f4-4fff-abb6-b261d3c5935b","Type":"ContainerStarted","Data":"cfef45861832497736e4dcb3ffcc3335671bebf8cb417ecf15be79981245e1d9"}
Jul 17 18:33:33.237496 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:33.237463    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2pb8h" event={"ID":"bd0f1bed-d043-4fbd-90d0-f3dfb22292de","Type":"ContainerStarted","Data":"2cb88ad5ae0aa723986c942fef21df2bce1d2c5b889ba9e3ca0de8df2144cf42"}
Jul 17 18:33:33.259757 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:33.259705    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2pb8h" podStartSLOduration=4.707639255 podStartE2EDuration="23.259686259s" totalImagesPullingTime="18.552047004s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:33.259099499 +0000 UTC m=+23.700608187" watchObservedRunningTime="2026-07-17 18:33:33.259686259 +0000 UTC m=+23.701194949"
Jul 17 18:33:33.260062 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:33.260031    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vtswf" podStartSLOduration=12.890206935 podStartE2EDuration="23.260022325s" totalImagesPullingTime="10.36981539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:32.359113744 +0000 UTC m=+22.800622434" watchObservedRunningTime="2026-07-17 18:33:33.260022325 +0000 UTC m=+23.701531015"
Jul 17 18:33:33.482055 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:33.482019    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vtswf"
Jul 17 18:33:33.482732 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:33.482710    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vtswf"
Jul 17 18:33:33.525584 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:33.525567    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 17 18:33:34.023168 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:34.023034    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T18:33:33.525581978Z","UUID":"e3109ad7-5140-4941-9f4f-588ac0077ce9","Handler":null,"Name":"","Endpoint":""}
Jul 17 18:33:34.024940 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:34.024916    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 17 18:33:34.025082 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:34.024964    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 17 18:33:34.119871 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:34.119840    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:34.119871 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:34.119857    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:34.120115 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:34.119976    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6ftw" podUID="b6eaddd6-5198-4d18-bd28-bd1d31df6994"
Jul 17 18:33:34.120115 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:34.120075    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:34.120115 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:34.120091    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt279" podUID="f9fc2de6-adf3-45c0-952a-e317f362aa51"
Jul 17 18:33:34.120272 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:34.120177    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nws8" podUID="1f8b093e-7ab9-406d-8a0e-c79e694cb205"
Jul 17 18:33:34.241980 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:34.241948    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz" event={"ID":"1eb72ece-f355-4599-b236-299ac5dce970","Type":"ContainerStarted","Data":"9dfe8061216a64124f0bf72decddcd06d3397d141dda937caffde7a50798bca1"}
Jul 17 18:33:34.242402 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:34.242341    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vtswf"
Jul 17 18:33:34.242713 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:34.242694    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vtswf"
Jul 17 18:33:35.245986 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:35.245950    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz" event={"ID":"1eb72ece-f355-4599-b236-299ac5dce970","Type":"ContainerStarted","Data":"217738622db40cf0e61ba3ae9cb76f0ce5ba141dac124781606b5e0471631d6c"}
Jul 17 18:33:35.248995 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:35.248975    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 18:33:35.249375 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:35.249355    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c" event={"ID":"6d169050-68f4-4fff-abb6-b261d3c5935b","Type":"ContainerStarted","Data":"a8503a9acf8fe2fa8372d6060dabc887a7ec0e44c2e9e5e40e53f4dd8e6abf7a"}
Jul 17 18:33:35.270477 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:35.270434    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvgdz" podStartSLOduration=4.234992529 podStartE2EDuration="25.27041956s" totalImagesPullingTime="21.035427031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:35.270143369 +0000 UTC m=+25.711652058" watchObservedRunningTime="2026-07-17 18:33:35.27041956 +0000 UTC m=+25.711928248"
Jul 17 18:33:36.119461 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:36.119182    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:36.119678 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:36.119181    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:36.119678 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:36.119553    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt279" podUID="f9fc2de6-adf3-45c0-952a-e317f362aa51"
Jul 17 18:33:36.119678 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:36.119182    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:36.119678 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:36.119624    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6ftw" podUID="b6eaddd6-5198-4d18-bd28-bd1d31df6994"
Jul 17 18:33:36.119914 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:36.119718    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nws8" podUID="1f8b093e-7ab9-406d-8a0e-c79e694cb205"
Jul 17 18:33:38.120230 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:38.120044    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:38.120996 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:38.120066    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:38.120996 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:38.120344    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt279" podUID="f9fc2de6-adf3-45c0-952a-e317f362aa51"
Jul 17 18:33:38.120996 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:38.120380    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6ftw" podUID="b6eaddd6-5198-4d18-bd28-bd1d31df6994"
Jul 17 18:33:38.120996 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:38.120087    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:38.120996 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:38.120487    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nws8" podUID="1f8b093e-7ab9-406d-8a0e-c79e694cb205"
Jul 17 18:33:38.259805 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:38.259767    2559 generic.go:356] "Generic (PLEG): container finished" podID="486589ec-aa0a-497e-88e6-79e52df71588" containerID="253a4aba3826ae4378c224a6abea33138a73fd29a71bf0116805be73bce961a6" exitCode=0
Jul 17 18:33:38.259965 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:38.259878    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cnj2q" event={"ID":"486589ec-aa0a-497e-88e6-79e52df71588","Type":"ContainerDied","Data":"253a4aba3826ae4378c224a6abea33138a73fd29a71bf0116805be73bce961a6"}
Jul 17 18:33:38.263590 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:38.263572    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 18:33:38.263931 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:38.263912    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c" event={"ID":"6d169050-68f4-4fff-abb6-b261d3c5935b","Type":"ContainerStarted","Data":"bce8facb0dcdf6673878780e6ab4339699e712b8247cdc6bf3dfa3ac90f4a26d"}
Jul 17 18:33:38.264110 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:38.264097    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:38.264170 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:38.264154    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:38.264341 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:38.264327    2559 scope.go:117] "RemoveContainer" containerID="b426b6ed87955b298a263a25cfa93888ee65c62a061a7d282e43a762dcda0aee"
Jul 17 18:33:38.280131 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:38.280112    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:39.270290 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:39.270218    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 18:33:39.270749 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:39.270628    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c" event={"ID":"6d169050-68f4-4fff-abb6-b261d3c5935b","Type":"ContainerStarted","Data":"bbe169f384b67ba3cba92d4dd632409b03c8cd820f218aa9c19537a0ce9e69df"}
Jul 17 18:33:39.270958 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:39.270940    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:39.273094 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:39.273066    2559 generic.go:356] "Generic (PLEG): container finished" podID="486589ec-aa0a-497e-88e6-79e52df71588" containerID="d7abdfd2eb9acaeb8872f5e9bd3d1a7cb4554ed961e76428b7604cb7be20dc90" exitCode=0
Jul 17 18:33:39.273197 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:39.273109    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cnj2q" event={"ID":"486589ec-aa0a-497e-88e6-79e52df71588","Type":"ContainerDied","Data":"d7abdfd2eb9acaeb8872f5e9bd3d1a7cb4554ed961e76428b7604cb7be20dc90"}
Jul 17 18:33:39.287823 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:39.287801    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:33:39.388123 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:39.388075    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c" podStartSLOduration=10.624112403 podStartE2EDuration="29.388060524s" totalImagesPullingTime="18.763948121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:39.331091008 +0000 UTC m=+29.772599697" watchObservedRunningTime="2026-07-17 18:33:39.388060524 +0000 UTC m=+29.829569213"
Jul 17 18:33:39.759878 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:39.759841    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5nws8"]
Jul 17 18:33:39.760023 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:39.759989    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:39.760085 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:39.760067    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nws8" podUID="1f8b093e-7ab9-406d-8a0e-c79e694cb205"
Jul 17 18:33:39.767944 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:39.767919    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m6ftw"]
Jul 17 18:33:39.768071 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:39.768050    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:39.768169 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:39.768145    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6ftw" podUID="b6eaddd6-5198-4d18-bd28-bd1d31df6994"
Jul 17 18:33:39.768733 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:39.768705    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tt279"]
Jul 17 18:33:39.768868 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:39.768854    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:39.768970 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:39.768946    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt279" podUID="f9fc2de6-adf3-45c0-952a-e317f362aa51"
Jul 17 18:33:40.277731 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:40.277497    2559 generic.go:356] "Generic (PLEG): container finished" podID="486589ec-aa0a-497e-88e6-79e52df71588" containerID="f3d11a4a64803d516ee790af76d8ab8dc073d865ae897f67a8965c980ed8607d" exitCode=0
Jul 17 18:33:40.278106 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:40.277578    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cnj2q" event={"ID":"486589ec-aa0a-497e-88e6-79e52df71588","Type":"ContainerDied","Data":"f3d11a4a64803d516ee790af76d8ab8dc073d865ae897f67a8965c980ed8607d"}
Jul 17 18:33:41.119779 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:41.119702    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:41.119920 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:41.119821    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:41.119920 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:41.119836    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6ftw" podUID="b6eaddd6-5198-4d18-bd28-bd1d31df6994"
Jul 17 18:33:41.120032 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:41.119934    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt279" podUID="f9fc2de6-adf3-45c0-952a-e317f362aa51"
Jul 17 18:33:41.120032 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:41.119991    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:41.120122 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:41.120046    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nws8" podUID="1f8b093e-7ab9-406d-8a0e-c79e694cb205"
Jul 17 18:33:43.119518 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:43.119479    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:43.120170 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:43.119485    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:43.120170 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:43.119698    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tt279" podUID="f9fc2de6-adf3-45c0-952a-e317f362aa51"
Jul 17 18:33:43.120170 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:43.119485    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:43.120170 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:43.119767    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m6ftw" podUID="b6eaddd6-5198-4d18-bd28-bd1d31df6994"
Jul 17 18:33:43.120170 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:43.119791    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5nws8" podUID="1f8b093e-7ab9-406d-8a0e-c79e694cb205"
Jul 17 18:33:43.704200 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:43.704168    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs\") pod \"network-metrics-daemon-tt279\" (UID: \"f9fc2de6-adf3-45c0-952a-e317f362aa51\") " pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:43.704471 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:43.704277    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:43.704471 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:43.704348    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs podName:f9fc2de6-adf3-45c0-952a-e317f362aa51 nodeName:}" failed. No retries permitted until 2026-07-17 18:34:15.704331198 +0000 UTC m=+66.145839866 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs") pod "network-metrics-daemon-tt279" (UID: "f9fc2de6-adf3-45c0-952a-e317f362aa51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:33:43.804526 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:43.804491    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpdgv\" (UniqueName: \"kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv\") pod \"network-check-target-m6ftw\" (UID: \"b6eaddd6-5198-4d18-bd28-bd1d31df6994\") " pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:43.804709 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:43.804678    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:33:43.804709 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:43.804703    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:33:43.804820 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:43.804715    2559 projected.go:196] Error preparing data for projected volume kube-api-access-cpdgv for pod openshift-network-diagnostics/network-check-target-m6ftw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:43.804820 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:43.804783    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv podName:b6eaddd6-5198-4d18-bd28-bd1d31df6994 nodeName:}" failed. No retries permitted until 2026-07-17 18:34:15.804764106 +0000 UTC m=+66.246272788 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cpdgv" (UniqueName: "kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv") pod "network-check-target-m6ftw" (UID: "b6eaddd6-5198-4d18-bd28-bd1d31df6994") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:33:43.905500 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:43.905461    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret\") pod \"global-pull-secret-syncer-5nws8\" (UID: \"1f8b093e-7ab9-406d-8a0e-c79e694cb205\") " pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:43.905690 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:43.905613    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:43.905690 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:43.905672    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret podName:1f8b093e-7ab9-406d-8a0e-c79e694cb205 nodeName:}" failed. No retries permitted until 2026-07-17 18:34:15.905656429 +0000 UTC m=+66.347165116 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret") pod "global-pull-secret-syncer-5nws8" (UID: "1f8b093e-7ab9-406d-8a0e-c79e694cb205") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:33:44.819296 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:44.819267    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-51.ec2.internal" event="NodeReady"
Jul 17 18:33:44.819670 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:44.819423    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 17 18:33:44.900283 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:44.900251    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lfvbt"]
Jul 17 18:33:44.902003 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:44.901988    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lfvbt"
Jul 17 18:33:44.903578 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:44.903555    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7drcz"]
Jul 17 18:33:44.905120 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:44.905103    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7drcz"
Jul 17 18:33:44.905581 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:44.905564    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 17 18:33:44.905667 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:44.905600    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 17 18:33:44.905799 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:44.905785    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lzjfd\""
Jul 17 18:33:44.905903 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:44.905887    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 17 18:33:44.908127 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:44.908108    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 17 18:33:44.909190 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:44.909127    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vz2rf\""
Jul 17 18:33:44.909286 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:44.909265    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 17 18:33:44.922984 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:44.922967    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lfvbt"]
Jul 17 18:33:44.925875 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:44.925856    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7drcz"]
Jul 17 18:33:45.013806 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.013770    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab87a034-e286-4b47-8513-9fb3c0b53084-tmp-dir\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:33:45.013806 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.013801    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ab87a034-e286-4b47-8513-9fb3c0b53084-config-volume\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:33:45.014045 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.013818    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:33:45.014045 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.013872    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl8zd\" (UniqueName: \"kubernetes.io/projected/ab87a034-e286-4b47-8513-9fb3c0b53084-kube-api-access-xl8zd\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:33:45.014045 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.013918    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0881b192-5909-49a5-ba16-65dbc6829189-cert\") pod \"ingress-canary-lfvbt\" (UID: \"0881b192-5909-49a5-ba16-65dbc6829189\") " pod="openshift-ingress-canary/ingress-canary-lfvbt"
Jul 17 18:33:45.014045 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.013941    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj2jn\" (UniqueName: \"kubernetes.io/projected/0881b192-5909-49a5-ba16-65dbc6829189-kube-api-access-vj2jn\") pod \"ingress-canary-lfvbt\" (UID: \"0881b192-5909-49a5-ba16-65dbc6829189\") " pod="openshift-ingress-canary/ingress-canary-lfvbt"
Jul 17 18:33:45.114953 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.114902    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab87a034-e286-4b47-8513-9fb3c0b53084-tmp-dir\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:33:45.114953 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.114953    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ab87a034-e286-4b47-8513-9fb3c0b53084-config-volume\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:33:45.115192 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.114976    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:33:45.115192 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.114996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl8zd\" (UniqueName: \"kubernetes.io/projected/ab87a034-e286-4b47-8513-9fb3c0b53084-kube-api-access-xl8zd\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:33:45.115192 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:45.115083    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:33:45.115192 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.115119    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0881b192-5909-49a5-ba16-65dbc6829189-cert\") pod \"ingress-canary-lfvbt\" (UID: \"0881b192-5909-49a5-ba16-65dbc6829189\") " pod="openshift-ingress-canary/ingress-canary-lfvbt"
Jul 17 18:33:45.115192 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:45.115155    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls podName:ab87a034-e286-4b47-8513-9fb3c0b53084 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:45.615138051 +0000 UTC m=+36.056646718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls") pod "dns-default-7drcz" (UID: "ab87a034-e286-4b47-8513-9fb3c0b53084") : secret "dns-default-metrics-tls" not found
Jul 17 18:33:45.115192 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.115178    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj2jn\" (UniqueName: \"kubernetes.io/projected/0881b192-5909-49a5-ba16-65dbc6829189-kube-api-access-vj2jn\") pod \"ingress-canary-lfvbt\" (UID: \"0881b192-5909-49a5-ba16-65dbc6829189\") " pod="openshift-ingress-canary/ingress-canary-lfvbt"
Jul 17 18:33:45.115460 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.115361    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab87a034-e286-4b47-8513-9fb3c0b53084-tmp-dir\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:33:45.115832 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.115814    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ab87a034-e286-4b47-8513-9fb3c0b53084-config-volume\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:33:45.119177 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.119151    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:33:45.119282 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.119255    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:33:45.119282 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.119279    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:33:45.119520 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.119474    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0881b192-5909-49a5-ba16-65dbc6829189-cert\") pod \"ingress-canary-lfvbt\" (UID: \"0881b192-5909-49a5-ba16-65dbc6829189\") " pod="openshift-ingress-canary/ingress-canary-lfvbt"
Jul 17 18:33:45.122119 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.122096    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 18:33:45.122250 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.122194    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 18:33:45.122332 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.122239    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 18:33:45.122390 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.122344    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 18:33:45.122441 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.122429    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-st5cn\""
Jul 17 18:33:45.122553 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.122537    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gbnj9\""
Jul 17 18:33:45.127329 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.127283    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl8zd\" (UniqueName: \"kubernetes.io/projected/ab87a034-e286-4b47-8513-9fb3c0b53084-kube-api-access-xl8zd\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:33:45.130122 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.130105    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj2jn\" (UniqueName: \"kubernetes.io/projected/0881b192-5909-49a5-ba16-65dbc6829189-kube-api-access-vj2jn\") pod \"ingress-canary-lfvbt\" (UID: \"0881b192-5909-49a5-ba16-65dbc6829189\") " pod="openshift-ingress-canary/ingress-canary-lfvbt"
Jul 17 18:33:45.212183 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.212087    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lfvbt"
Jul 17 18:33:45.293366 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.293171    2559 generic.go:356] "Generic (PLEG): container finished" podID="486589ec-aa0a-497e-88e6-79e52df71588" containerID="1d2ead4d0b02814245cd7a4923b06b6400f6deadbe050bb877da1b10de27f510" exitCode=0
Jul 17 18:33:45.293366 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.293249    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cnj2q" event={"ID":"486589ec-aa0a-497e-88e6-79e52df71588","Type":"ContainerDied","Data":"1d2ead4d0b02814245cd7a4923b06b6400f6deadbe050bb877da1b10de27f510"}
Jul 17 18:33:45.354761 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.354659    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lfvbt"]
Jul 17 18:33:45.358722 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:45.358695    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0881b192_5909_49a5_ba16_65dbc6829189.slice/crio-964eacf1fcee0b8fa5630728687cb5e8fc0aa49d9da927bab29f48bc6a09fa46 WatchSource:0}: Error finding container 964eacf1fcee0b8fa5630728687cb5e8fc0aa49d9da927bab29f48bc6a09fa46: Status 404 returned error can't find the container with id 964eacf1fcee0b8fa5630728687cb5e8fc0aa49d9da927bab29f48bc6a09fa46
Jul 17 18:33:45.618505 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:45.618470    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:33:45.618657 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:45.618623    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:33:45.618696 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:45.618686    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls podName:ab87a034-e286-4b47-8513-9fb3c0b53084 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:46.618668928 +0000 UTC m=+37.060177612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls") pod "dns-default-7drcz" (UID: "ab87a034-e286-4b47-8513-9fb3c0b53084") : secret "dns-default-metrics-tls" not found
Jul 17 18:33:46.299175 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:46.299138    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lfvbt" event={"ID":"0881b192-5909-49a5-ba16-65dbc6829189","Type":"ContainerStarted","Data":"964eacf1fcee0b8fa5630728687cb5e8fc0aa49d9da927bab29f48bc6a09fa46"}
Jul 17 18:33:46.301925 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:46.301901    2559 generic.go:356] "Generic (PLEG): container finished" podID="486589ec-aa0a-497e-88e6-79e52df71588" containerID="138df1b38aa66d89765d3797dac66a6ff4d73c1efb5700179625ac601be757ce" exitCode=0
Jul 17 18:33:46.302031 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:46.301936    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cnj2q" event={"ID":"486589ec-aa0a-497e-88e6-79e52df71588","Type":"ContainerDied","Data":"138df1b38aa66d89765d3797dac66a6ff4d73c1efb5700179625ac601be757ce"}
Jul 17 18:33:46.625802 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:46.625766    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:33:46.626015 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:46.625992    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:33:46.626100 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:46.626089    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls podName:ab87a034-e286-4b47-8513-9fb3c0b53084 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:48.626070697 +0000 UTC m=+39.067579384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls") pod "dns-default-7drcz" (UID: "ab87a034-e286-4b47-8513-9fb3c0b53084") : secret "dns-default-metrics-tls" not found
Jul 17 18:33:47.308212 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:47.308170    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cnj2q" event={"ID":"486589ec-aa0a-497e-88e6-79e52df71588","Type":"ContainerStarted","Data":"74ce2c8ff7cb2da2280d782578335c7f56e0736944e32a9a1a254dbcd131784e"}
Jul 17 18:33:47.356115 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:47.355972    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cnj2q" podStartSLOduration=7.683148195 podStartE2EDuration="37.355951965s" totalImagesPullingTime="29.67280377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:47.355949163 +0000 UTC m=+37.797457854" watchObservedRunningTime="2026-07-17 18:33:47.355951965 +0000 UTC m=+37.797460656"
Jul 17 18:33:48.312804 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:48.312636    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lfvbt" event={"ID":"0881b192-5909-49a5-ba16-65dbc6829189","Type":"ContainerStarted","Data":"637c8a5053b481d54a01a859b37554674745d580ba38308d556af01bb6274f81"}
Jul 17 18:33:48.347169 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:48.347123    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lfvbt" podStartSLOduration=1.663001658 podStartE2EDuration="4.347108339s" totalImagesPullingTime="2.684106681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:48.347040718 +0000 UTC m=+38.788549406" watchObservedRunningTime="2026-07-17 18:33:48.347108339 +0000 UTC m=+38.788617028"
Jul 17 18:33:48.641810 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:48.641724    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:33:48.641941 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:48.641873    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:33:48.641980 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:48.641951    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls podName:ab87a034-e286-4b47-8513-9fb3c0b53084 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:52.641932205 +0000 UTC m=+43.083440892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls") pod "dns-default-7drcz" (UID: "ab87a034-e286-4b47-8513-9fb3c0b53084") : secret "dns-default-metrics-tls" not found
Jul 17 18:33:52.670481 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:52.670440    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:33:52.670936 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:52.670596    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:33:52.670936 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:33:52.670659    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls podName:ab87a034-e286-4b47-8513-9fb3c0b53084 nodeName:}" failed. No retries permitted until 2026-07-17 18:34:00.670644511 +0000 UTC m=+51.112153177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls") pod "dns-default-7drcz" (UID: "ab87a034-e286-4b47-8513-9fb3c0b53084") : secret "dns-default-metrics-tls" not found
Jul 17 18:33:58.782059 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:58.782021    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8"]
Jul 17 18:33:58.795235 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:58.795213    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8"
Jul 17 18:33:58.801786 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:58.801698    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 17 18:33:58.801865 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:58.801805    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 17 18:33:58.801986 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:58.801940    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-7kbx5\""
Jul 17 18:33:58.805099 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:58.805077    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 17 18:33:58.805194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:58.805169    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 17 18:33:58.809174 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:58.809155    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8"]
Jul 17 18:33:58.914424 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:58.914387    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmhfd\" (UniqueName: \"kubernetes.io/projected/5489e8dd-e1cb-4247-88a4-4c2d44f0633c-kube-api-access-qmhfd\") pod \"managed-serviceaccount-addon-agent-6fd976888b-6q6q8\" (UID: \"5489e8dd-e1cb-4247-88a4-4c2d44f0633c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8"
Jul 17 18:33:58.914424 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:58.914428    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5489e8dd-e1cb-4247-88a4-4c2d44f0633c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fd976888b-6q6q8\" (UID: \"5489e8dd-e1cb-4247-88a4-4c2d44f0633c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8"
Jul 17 18:33:59.015185 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:59.015149    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmhfd\" (UniqueName: \"kubernetes.io/projected/5489e8dd-e1cb-4247-88a4-4c2d44f0633c-kube-api-access-qmhfd\") pod \"managed-serviceaccount-addon-agent-6fd976888b-6q6q8\" (UID: \"5489e8dd-e1cb-4247-88a4-4c2d44f0633c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8"
Jul 17 18:33:59.015276 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:59.015210    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5489e8dd-e1cb-4247-88a4-4c2d44f0633c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fd976888b-6q6q8\" (UID: \"5489e8dd-e1cb-4247-88a4-4c2d44f0633c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8"
Jul 17 18:33:59.019552 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:59.019517    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5489e8dd-e1cb-4247-88a4-4c2d44f0633c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fd976888b-6q6q8\" (UID: \"5489e8dd-e1cb-4247-88a4-4c2d44f0633c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8"
Jul 17 18:33:59.026523 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:59.026499    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmhfd\" (UniqueName: \"kubernetes.io/projected/5489e8dd-e1cb-4247-88a4-4c2d44f0633c-kube-api-access-qmhfd\") pod \"managed-serviceaccount-addon-agent-6fd976888b-6q6q8\" (UID: \"5489e8dd-e1cb-4247-88a4-4c2d44f0633c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8"
Jul 17 18:33:59.119867 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:59.119838    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8"
Jul 17 18:33:59.272605 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:59.272569    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8"]
Jul 17 18:33:59.276164 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:33:59.276139    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5489e8dd_e1cb_4247_88a4_4c2d44f0633c.slice/crio-40107c591416f8f895088bd581e14afdcb213b40534c5719074ceb232cff4a37 WatchSource:0}: Error finding container 40107c591416f8f895088bd581e14afdcb213b40534c5719074ceb232cff4a37: Status 404 returned error can't find the container with id 40107c591416f8f895088bd581e14afdcb213b40534c5719074ceb232cff4a37
Jul 17 18:33:59.338824 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:33:59.338791    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8" event={"ID":"5489e8dd-e1cb-4247-88a4-4c2d44f0633c","Type":"ContainerStarted","Data":"40107c591416f8f895088bd581e14afdcb213b40534c5719074ceb232cff4a37"}
Jul 17 18:34:00.727777 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:00.727734    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:34:00.728240 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:34:00.727849    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:34:00.728240 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:34:00.727901    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls podName:ab87a034-e286-4b47-8513-9fb3c0b53084 nodeName:}" failed. No retries permitted until 2026-07-17 18:34:16.727887327 +0000 UTC m=+67.169395994 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls") pod "dns-default-7drcz" (UID: "ab87a034-e286-4b47-8513-9fb3c0b53084") : secret "dns-default-metrics-tls" not found
Jul 17 18:34:07.359876 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:07.359836    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8" event={"ID":"5489e8dd-e1cb-4247-88a4-4c2d44f0633c","Type":"ContainerStarted","Data":"ac5641a621a90624a790a92cdc1d344a3f8a0f5dbde92978b248c3e8ea4f6007"}
Jul 17 18:34:07.403733 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:07.403686    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8" podStartSLOduration=1.881075709 podStartE2EDuration="9.403672293s" totalImagesPullingTime="7.522596584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:34:07.403603165 +0000 UTC m=+57.845111853" watchObservedRunningTime="2026-07-17 18:34:07.403672293 +0000 UTC m=+57.845181031"
Jul 17 18:34:11.292472 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:11.292445    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xhn5c"
Jul 17 18:34:15.730959 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:15.730916    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs\") pod \"network-metrics-daemon-tt279\" (UID: \"f9fc2de6-adf3-45c0-952a-e317f362aa51\") " pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:34:15.733819 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:15.733801    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 18:34:15.741187 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:34:15.741170    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 17 18:34:15.741278 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:34:15.741268    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs podName:f9fc2de6-adf3-45c0-952a-e317f362aa51 nodeName:}" failed. No retries permitted until 2026-07-17 18:35:19.741248897 +0000 UTC m=+130.182757564 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs") pod "network-metrics-daemon-tt279" (UID: "f9fc2de6-adf3-45c0-952a-e317f362aa51") : secret "metrics-daemon-secret" not found
Jul 17 18:34:15.832260 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:15.832211    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpdgv\" (UniqueName: \"kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv\") pod \"network-check-target-m6ftw\" (UID: \"b6eaddd6-5198-4d18-bd28-bd1d31df6994\") " pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:34:15.835067 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:15.835049    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 18:34:15.845377 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:15.845359    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 18:34:15.855950 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:15.855921    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpdgv\" (UniqueName: \"kubernetes.io/projected/b6eaddd6-5198-4d18-bd28-bd1d31df6994-kube-api-access-cpdgv\") pod \"network-check-target-m6ftw\" (UID: \"b6eaddd6-5198-4d18-bd28-bd1d31df6994\") " pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:34:15.932685 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:15.932651    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret\") pod \"global-pull-secret-syncer-5nws8\" (UID: \"1f8b093e-7ab9-406d-8a0e-c79e694cb205\") " pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:34:15.935691 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:15.935674    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 18:34:15.945025 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:15.945001    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f8b093e-7ab9-406d-8a0e-c79e694cb205-original-pull-secret\") pod \"global-pull-secret-syncer-5nws8\" (UID: \"1f8b093e-7ab9-406d-8a0e-c79e694cb205\") " pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:34:16.039396 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:16.039320    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5nws8"
Jul 17 18:34:16.047786 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:16.047764    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-st5cn\""
Jul 17 18:34:16.055525 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:16.055502    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:34:16.189363 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:16.189331    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5nws8"]
Jul 17 18:34:16.193006 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:34:16.192978    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8b093e_7ab9_406d_8a0e_c79e694cb205.slice/crio-c718b188d97f2cba6441246af0e5df193cf904d2d0e3c90b303d15a123d2952a WatchSource:0}: Error finding container c718b188d97f2cba6441246af0e5df193cf904d2d0e3c90b303d15a123d2952a: Status 404 returned error can't find the container with id c718b188d97f2cba6441246af0e5df193cf904d2d0e3c90b303d15a123d2952a
Jul 17 18:34:16.211136 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:16.211113    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m6ftw"]
Jul 17 18:34:16.213661 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:34:16.213638    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6eaddd6_5198_4d18_bd28_bd1d31df6994.slice/crio-d80117f9b483ad38b4ee136ee14bd61fa8de15e41584457c9172f13db298cf25 WatchSource:0}: Error finding container d80117f9b483ad38b4ee136ee14bd61fa8de15e41584457c9172f13db298cf25: Status 404 returned error can't find the container with id d80117f9b483ad38b4ee136ee14bd61fa8de15e41584457c9172f13db298cf25
Jul 17 18:34:16.381918 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:16.381881    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5nws8" event={"ID":"1f8b093e-7ab9-406d-8a0e-c79e694cb205","Type":"ContainerStarted","Data":"c718b188d97f2cba6441246af0e5df193cf904d2d0e3c90b303d15a123d2952a"}
Jul 17 18:34:16.382960 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:16.382928    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m6ftw" event={"ID":"b6eaddd6-5198-4d18-bd28-bd1d31df6994","Type":"ContainerStarted","Data":"d80117f9b483ad38b4ee136ee14bd61fa8de15e41584457c9172f13db298cf25"}
Jul 17 18:34:16.738542 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:16.738446    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:34:16.738957 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:34:16.738608    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:34:16.738957 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:34:16.738691    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls podName:ab87a034-e286-4b47-8513-9fb3c0b53084 nodeName:}" failed. No retries permitted until 2026-07-17 18:34:48.738665836 +0000 UTC m=+99.180174522 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls") pod "dns-default-7drcz" (UID: "ab87a034-e286-4b47-8513-9fb3c0b53084") : secret "dns-default-metrics-tls" not found
Jul 17 18:34:21.400513 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:21.400461    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m6ftw" event={"ID":"b6eaddd6-5198-4d18-bd28-bd1d31df6994","Type":"ContainerStarted","Data":"d5ea8bdc835b856e408ecbeb0dc77f6b735b0a1f8d193cd31e765496439d6d33"}
Jul 17 18:34:21.401009 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:21.400523    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:34:21.401757 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:21.401730    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5nws8" event={"ID":"1f8b093e-7ab9-406d-8a0e-c79e694cb205","Type":"ContainerStarted","Data":"27630ca843f9a1ea86bd78188c009cf5c5073689f5bde5ec6d2dd75fa4058c80"}
Jul 17 18:34:21.450770 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:21.450722    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m6ftw" podStartSLOduration=66.771405201 podStartE2EDuration="1m11.450707026s" totalImagesPullingTime="4.679301825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:34:21.431125609 +0000 UTC m=+71.872634309" watchObservedRunningTime="2026-07-17 18:34:21.450707026 +0000 UTC m=+71.892215734"
Jul 17 18:34:48.743405 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:48.743260    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:34:48.743405 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:34:48.743397    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:34:48.743897 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:34:48.743467    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls podName:ab87a034-e286-4b47-8513-9fb3c0b53084 nodeName:}" failed. No retries permitted until 2026-07-17 18:35:52.74345172 +0000 UTC m=+163.184960387 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls") pod "dns-default-7drcz" (UID: "ab87a034-e286-4b47-8513-9fb3c0b53084") : secret "dns-default-metrics-tls" not found
Jul 17 18:34:52.406516 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:52.406490    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m6ftw"
Jul 17 18:34:52.423458 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:34:52.423411    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5nws8" podStartSLOduration=97.719275244 podStartE2EDuration="1m42.423376969s" totalImagesPullingTime="4.704101725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:34:21.451280561 +0000 UTC m=+71.892789249" watchObservedRunningTime="2026-07-17 18:34:52.423376969 +0000 UTC m=+102.864885651"
Jul 17 18:35:19.748526 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:35:19.748471    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs\") pod \"network-metrics-daemon-tt279\" (UID: \"f9fc2de6-adf3-45c0-952a-e317f362aa51\") " pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:35:19.749033 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:35:19.748611    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 17 18:35:19.749033 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:35:19.748669    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs podName:f9fc2de6-adf3-45c0-952a-e317f362aa51 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:21.748653164 +0000 UTC m=+252.190161830 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs") pod "network-metrics-daemon-tt279" (UID: "f9fc2de6-adf3-45c0-952a-e317f362aa51") : secret "metrics-daemon-secret" not found
Jul 17 18:35:47.917164 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:35:47.917104    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7drcz" podUID="ab87a034-e286-4b47-8513-9fb3c0b53084"
Jul 17 18:35:48.150381 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:35:48.150335    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-tt279" podUID="f9fc2de6-adf3-45c0-952a-e317f362aa51"
Jul 17 18:35:48.606689 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:35:48.606658    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7drcz"
Jul 17 18:35:52.763637 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:35:52.763596    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:35:52.764024 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:35:52.763757    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:35:52.764024 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:35:52.763833    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls podName:ab87a034-e286-4b47-8513-9fb3c0b53084 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:54.763816801 +0000 UTC m=+285.205325467 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls") pod "dns-default-7drcz" (UID: "ab87a034-e286-4b47-8513-9fb3c0b53084") : secret "dns-default-metrics-tls" not found
Jul 17 18:36:00.121109 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:36:00.121026    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:36:07.655528 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:36:07.655487    2559 generic.go:356] "Generic (PLEG): container finished" podID="5489e8dd-e1cb-4247-88a4-4c2d44f0633c" containerID="ac5641a621a90624a790a92cdc1d344a3f8a0f5dbde92978b248c3e8ea4f6007" exitCode=255
Jul 17 18:36:07.655952 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:36:07.655562    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8" event={"ID":"5489e8dd-e1cb-4247-88a4-4c2d44f0633c","Type":"ContainerDied","Data":"ac5641a621a90624a790a92cdc1d344a3f8a0f5dbde92978b248c3e8ea4f6007"}
Jul 17 18:36:07.660718 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:36:07.660703    2559 scope.go:117] "RemoveContainer" containerID="ac5641a621a90624a790a92cdc1d344a3f8a0f5dbde92978b248c3e8ea4f6007"
Jul 17 18:36:08.660275 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:36:08.660238    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8" event={"ID":"5489e8dd-e1cb-4247-88a4-4c2d44f0633c","Type":"ContainerStarted","Data":"f3d3cabe49a07a2c92b79e141f8d4fb62390981c7f1d7b4f0c8e091903253c43"}
Jul 17 18:37:16.666762 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.666721    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-vv4vz"]
Jul 17 18:37:16.668789 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.668762    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l"]
Jul 17 18:37:16.668917 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.668895    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-vv4vz"
Jul 17 18:37:16.670625 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.670599    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l"
Jul 17 18:37:16.671905 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.671881    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xrdrx\""
Jul 17 18:37:16.671905 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.671890    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 17 18:37:16.672102 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.671932    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 17 18:37:16.672213 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.672192    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 17 18:37:16.672388 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.672372    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 17 18:37:16.672475 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.672417    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks5dn"]
Jul 17 18:37:16.674004 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.673983    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks5dn"
Jul 17 18:37:16.674183 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.674165    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 17 18:37:16.674183 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.674180    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 17 18:37:16.674295 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.674189    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6xm9m\""
Jul 17 18:37:16.674295 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.674228    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 17 18:37:16.674295 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.674286    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 17 18:37:16.677181 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.677162    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 17 18:37:16.677181 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.677171    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 17 18:37:16.677365 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.677344    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 17 18:37:16.677986 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.677969    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-l2btk\""
Jul 17 18:37:16.683197 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.683177    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 17 18:37:16.688278 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.687958    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l"]
Jul 17 18:37:16.689119 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.689099    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-vv4vz"]
Jul 17 18:37:16.699758 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.699738    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks5dn"]
Jul 17 18:37:16.722892 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.722863    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5bd5031-5dc3-446b-88ea-83821a9f3faf-config\") pod \"console-operator-565b7b569d-vv4vz\" (UID: \"b5bd5031-5dc3-446b-88ea-83821a9f3faf\") " pod="openshift-console-operator/console-operator-565b7b569d-vv4vz"
Jul 17 18:37:16.722892 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.722893    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-978kz\" (UniqueName: \"kubernetes.io/projected/b5bd5031-5dc3-446b-88ea-83821a9f3faf-kube-api-access-978kz\") pod \"console-operator-565b7b569d-vv4vz\" (UID: \"b5bd5031-5dc3-446b-88ea-83821a9f3faf\") " pod="openshift-console-operator/console-operator-565b7b569d-vv4vz"
Jul 17 18:37:16.723076 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.722924    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrsjn\" (UniqueName: \"kubernetes.io/projected/39a38b78-ab33-411b-9a10-4cd932d6c535-kube-api-access-qrsjn\") pod \"cluster-monitoring-operator-76cf8966f8-xhz5l\" (UID: \"39a38b78-ab33-411b-9a10-4cd932d6c535\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l"
Jul 17 18:37:16.723076 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.722972    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c468674e-6f4b-4a05-abf6-87624054ac4c-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ks5dn\" (UID: \"c468674e-6f4b-4a05-abf6-87624054ac4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks5dn"
Jul 17 18:37:16.723076 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.723010    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5bd5031-5dc3-446b-88ea-83821a9f3faf-trusted-ca\") pod \"console-operator-565b7b569d-vv4vz\" (UID: \"b5bd5031-5dc3-446b-88ea-83821a9f3faf\") " pod="openshift-console-operator/console-operator-565b7b569d-vv4vz"
Jul 17 18:37:16.723076 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.723030    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/39a38b78-ab33-411b-9a10-4cd932d6c535-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-xhz5l\" (UID: \"39a38b78-ab33-411b-9a10-4cd932d6c535\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l"
Jul 17 18:37:16.723076 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.723065    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b5bd5031-5dc3-446b-88ea-83821a9f3faf-serving-cert\") pod \"console-operator-565b7b569d-vv4vz\" (UID: \"b5bd5031-5dc3-446b-88ea-83821a9f3faf\") " pod="openshift-console-operator/console-operator-565b7b569d-vv4vz"
Jul 17 18:37:16.723227 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.723080    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd62t\" (UniqueName: \"kubernetes.io/projected/c468674e-6f4b-4a05-abf6-87624054ac4c-kube-api-access-rd62t\") pod \"cluster-samples-operator-79b7c55f87-ks5dn\" (UID: \"c468674e-6f4b-4a05-abf6-87624054ac4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks5dn"
Jul 17 18:37:16.723227 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.723098    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-xhz5l\" (UID: \"39a38b78-ab33-411b-9a10-4cd932d6c535\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l"
Jul 17 18:37:16.763997 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.763964    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-tmwrz"]
Jul 17 18:37:16.765851 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.765834    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-tmwrz"
Jul 17 18:37:16.768934 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.768905    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-thxzx"]
Jul 17 18:37:16.769814 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.769779    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 17 18:37:16.770784 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.770767    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-thxzx"
Jul 17 18:37:16.771531 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.771516    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2gmkb\""
Jul 17 18:37:16.771696 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.771677    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 17 18:37:16.771860 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.771835    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 17 18:37:16.771961 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.771945    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 17 18:37:16.774942 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.774924    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 17 18:37:16.778261 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.778182    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 17 18:37:16.778261 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.778212    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 17 18:37:16.778523 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.778452    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 17 18:37:16.778523 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.778466    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-6sxlh\""
Jul 17 18:37:16.778523 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.778455    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 17 18:37:16.782171 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.782154    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-tmwrz"]
Jul 17 18:37:16.790122 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.790101    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-thxzx"]
Jul 17 18:37:16.823395 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.823360    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5bd5031-5dc3-446b-88ea-83821a9f3faf-config\") pod \"console-operator-565b7b569d-vv4vz\" (UID: \"b5bd5031-5dc3-446b-88ea-83821a9f3faf\") " pod="openshift-console-operator/console-operator-565b7b569d-vv4vz"
Jul 17 18:37:16.823395 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.823395    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-978kz\" (UniqueName: \"kubernetes.io/projected/b5bd5031-5dc3-446b-88ea-83821a9f3faf-kube-api-access-978kz\") pod \"console-operator-565b7b569d-vv4vz\" (UID: \"b5bd5031-5dc3-446b-88ea-83821a9f3faf\") " pod="openshift-console-operator/console-operator-565b7b569d-vv4vz"
Jul 17 18:37:16.823606 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.823423    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k9d9\" (UniqueName: \"kubernetes.io/projected/f86765ef-cbaf-4c55-aa42-8451d8e07363-kube-api-access-9k9d9\") pod \"service-ca-operator-5bcbc94c78-thxzx\" (UID: \"f86765ef-cbaf-4c55-aa42-8451d8e07363\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-thxzx"
Jul 17 18:37:16.823606 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.823452    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f86765ef-cbaf-4c55-aa42-8451d8e07363-config\") pod \"service-ca-operator-5bcbc94c78-thxzx\" (UID: \"f86765ef-cbaf-4c55-aa42-8451d8e07363\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-thxzx"
Jul 17 18:37:16.823606 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.823517    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b92733f1-3721-44de-a7d9-c19deb63d6a4-snapshots\") pod \"insights-operator-555d9d7885-tmwrz\" (UID: \"b92733f1-3721-44de-a7d9-c19deb63d6a4\") " pod="openshift-insights/insights-operator-555d9d7885-tmwrz"
Jul 17 18:37:16.823606 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.823554    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b92733f1-3721-44de-a7d9-c19deb63d6a4-service-ca-bundle\") pod \"insights-operator-555d9d7885-tmwrz\" (UID: \"b92733f1-3721-44de-a7d9-c19deb63d6a4\") " pod="openshift-insights/insights-operator-555d9d7885-tmwrz"
Jul 17 18:37:16.823606 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.823577    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f86765ef-cbaf-4c55-aa42-8451d8e07363-serving-cert\") pod \"service-ca-operator-5bcbc94c78-thxzx\" (UID: \"f86765ef-cbaf-4c55-aa42-8451d8e07363\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-thxzx"
Jul 17 18:37:16.823760 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.823610    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrsjn\" (UniqueName: \"kubernetes.io/projected/39a38b78-ab33-411b-9a10-4cd932d6c535-kube-api-access-qrsjn\") pod \"cluster-monitoring-operator-76cf8966f8-xhz5l\" (UID: \"39a38b78-ab33-411b-9a10-4cd932d6c535\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l"
Jul 17 18:37:16.823760 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.823638    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c468674e-6f4b-4a05-abf6-87624054ac4c-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ks5dn\" (UID: \"c468674e-6f4b-4a05-abf6-87624054ac4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks5dn"
Jul 17 18:37:16.823760 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.823679    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5bd5031-5dc3-446b-88ea-83821a9f3faf-trusted-ca\") pod \"console-operator-565b7b569d-vv4vz\" (UID: \"b5bd5031-5dc3-446b-88ea-83821a9f3faf\") " pod="openshift-console-operator/console-operator-565b7b569d-vv4vz"
Jul 17 18:37:16.823760 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.823704    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/39a38b78-ab33-411b-9a10-4cd932d6c535-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-xhz5l\" (UID: \"39a38b78-ab33-411b-9a10-4cd932d6c535\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l"
Jul 17 18:37:16.823760 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.823739    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b92733f1-3721-44de-a7d9-c19deb63d6a4-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-tmwrz\" (UID: \"b92733f1-3721-44de-a7d9-c19deb63d6a4\") " pod="openshift-insights/insights-operator-555d9d7885-tmwrz"
Jul 17 18:37:16.823760 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.823759    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ttz9\" (UniqueName: \"kubernetes.io/projected/b92733f1-3721-44de-a7d9-c19deb63d6a4-kube-api-access-2ttz9\") pod \"insights-operator-555d9d7885-tmwrz\" (UID: \"b92733f1-3721-44de-a7d9-c19deb63d6a4\") " pod="openshift-insights/insights-operator-555d9d7885-tmwrz"
Jul 17 18:37:16.824044 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:16.823778    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 18:37:16.824044 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.823818    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b5bd5031-5dc3-446b-88ea-83821a9f3faf-serving-cert\") pod \"console-operator-565b7b569d-vv4vz\" (UID: \"b5bd5031-5dc3-446b-88ea-83821a9f3faf\") " pod="openshift-console-operator/console-operator-565b7b569d-vv4vz"
Jul 17 18:37:16.824044 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.823849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd62t\" (UniqueName: \"kubernetes.io/projected/c468674e-6f4b-4a05-abf6-87624054ac4c-kube-api-access-rd62t\") pod \"cluster-samples-operator-79b7c55f87-ks5dn\" (UID: \"c468674e-6f4b-4a05-abf6-87624054ac4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks5dn"
Jul 17 18:37:16.824044 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:16.823860    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c468674e-6f4b-4a05-abf6-87624054ac4c-samples-operator-tls podName:c468674e-6f4b-4a05-abf6-87624054ac4c nodeName:}" failed. No retries permitted until 2026-07-17 18:37:17.323840224 +0000 UTC m=+247.765348905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c468674e-6f4b-4a05-abf6-87624054ac4c-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ks5dn" (UID: "c468674e-6f4b-4a05-abf6-87624054ac4c") : secret "samples-operator-tls" not found
Jul 17 18:37:16.824044 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.823892    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b92733f1-3721-44de-a7d9-c19deb63d6a4-tmp\") pod \"insights-operator-555d9d7885-tmwrz\" (UID: \"b92733f1-3721-44de-a7d9-c19deb63d6a4\") " pod="openshift-insights/insights-operator-555d9d7885-tmwrz"
Jul 17 18:37:16.824044 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.823921    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b92733f1-3721-44de-a7d9-c19deb63d6a4-serving-cert\") pod \"insights-operator-555d9d7885-tmwrz\" (UID: \"b92733f1-3721-44de-a7d9-c19deb63d6a4\") " pod="openshift-insights/insights-operator-555d9d7885-tmwrz"
Jul 17 18:37:16.824044 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.823958    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-xhz5l\" (UID: \"39a38b78-ab33-411b-9a10-4cd932d6c535\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l"
Jul 17 18:37:16.824438 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:16.824095    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 18:37:16.824438 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:16.824145    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls podName:39a38b78-ab33-411b-9a10-4cd932d6c535 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:17.32413114 +0000 UTC m=+247.765639807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-xhz5l" (UID: "39a38b78-ab33-411b-9a10-4cd932d6c535") : secret "cluster-monitoring-operator-tls" not found
Jul 17 18:37:16.824438 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.824152    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5bd5031-5dc3-446b-88ea-83821a9f3faf-config\") pod \"console-operator-565b7b569d-vv4vz\" (UID: \"b5bd5031-5dc3-446b-88ea-83821a9f3faf\") " pod="openshift-console-operator/console-operator-565b7b569d-vv4vz"
Jul 17 18:37:16.824661 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.824462    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5bd5031-5dc3-446b-88ea-83821a9f3faf-trusted-ca\") pod \"console-operator-565b7b569d-vv4vz\" (UID: \"b5bd5031-5dc3-446b-88ea-83821a9f3faf\") " pod="openshift-console-operator/console-operator-565b7b569d-vv4vz"
Jul 17 18:37:16.824661 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.824655    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/39a38b78-ab33-411b-9a10-4cd932d6c535-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-xhz5l\" (UID: \"39a38b78-ab33-411b-9a10-4cd932d6c535\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l"
Jul 17 18:37:16.826459 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.826436    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b5bd5031-5dc3-446b-88ea-83821a9f3faf-serving-cert\") pod \"console-operator-565b7b569d-vv4vz\" (UID: \"b5bd5031-5dc3-446b-88ea-83821a9f3faf\") " pod="openshift-console-operator/console-operator-565b7b569d-vv4vz"
Jul 17 18:37:16.834027 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.834000    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-978kz\" (UniqueName: \"kubernetes.io/projected/b5bd5031-5dc3-446b-88ea-83821a9f3faf-kube-api-access-978kz\") pod \"console-operator-565b7b569d-vv4vz\" (UID: \"b5bd5031-5dc3-446b-88ea-83821a9f3faf\") " pod="openshift-console-operator/console-operator-565b7b569d-vv4vz"
Jul 17 18:37:16.834133 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.834094    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrsjn\" (UniqueName: \"kubernetes.io/projected/39a38b78-ab33-411b-9a10-4cd932d6c535-kube-api-access-qrsjn\") pod \"cluster-monitoring-operator-76cf8966f8-xhz5l\" (UID: \"39a38b78-ab33-411b-9a10-4cd932d6c535\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l"
Jul 17 18:37:16.834185 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.834131    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd62t\" (UniqueName: \"kubernetes.io/projected/c468674e-6f4b-4a05-abf6-87624054ac4c-kube-api-access-rd62t\") pod \"cluster-samples-operator-79b7c55f87-ks5dn\" (UID: \"c468674e-6f4b-4a05-abf6-87624054ac4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks5dn"
Jul 17 18:37:16.925175 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.925085    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b92733f1-3721-44de-a7d9-c19deb63d6a4-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-tmwrz\" (UID: \"b92733f1-3721-44de-a7d9-c19deb63d6a4\") " pod="openshift-insights/insights-operator-555d9d7885-tmwrz"
Jul 17 18:37:16.925175 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.925118    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ttz9\" (UniqueName: \"kubernetes.io/projected/b92733f1-3721-44de-a7d9-c19deb63d6a4-kube-api-access-2ttz9\") pod \"insights-operator-555d9d7885-tmwrz\" (UID: \"b92733f1-3721-44de-a7d9-c19deb63d6a4\") " pod="openshift-insights/insights-operator-555d9d7885-tmwrz"
Jul 17 18:37:16.925175 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.925159    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b92733f1-3721-44de-a7d9-c19deb63d6a4-tmp\") pod \"insights-operator-555d9d7885-tmwrz\" (UID: \"b92733f1-3721-44de-a7d9-c19deb63d6a4\") " pod="openshift-insights/insights-operator-555d9d7885-tmwrz"
Jul 17 18:37:16.925175 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.925177    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b92733f1-3721-44de-a7d9-c19deb63d6a4-serving-cert\") pod \"insights-operator-555d9d7885-tmwrz\" (UID: \"b92733f1-3721-44de-a7d9-c19deb63d6a4\") " pod="openshift-insights/insights-operator-555d9d7885-tmwrz"
Jul 17 18:37:16.925547 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.925215    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k9d9\" (UniqueName: \"kubernetes.io/projected/f86765ef-cbaf-4c55-aa42-8451d8e07363-kube-api-access-9k9d9\") pod \"service-ca-operator-5bcbc94c78-thxzx\" (UID: \"f86765ef-cbaf-4c55-aa42-8451d8e07363\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-thxzx"
Jul 17 18:37:16.925547 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.925231    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f86765ef-cbaf-4c55-aa42-8451d8e07363-config\") pod \"service-ca-operator-5bcbc94c78-thxzx\" (UID: \"f86765ef-cbaf-4c55-aa42-8451d8e07363\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-thxzx"
Jul 17 18:37:16.925547 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.925258    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b92733f1-3721-44de-a7d9-c19deb63d6a4-snapshots\") pod \"insights-operator-555d9d7885-tmwrz\" (UID: \"b92733f1-3721-44de-a7d9-c19deb63d6a4\") " pod="openshift-insights/insights-operator-555d9d7885-tmwrz"
Jul 17 18:37:16.925547 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.925281    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b92733f1-3721-44de-a7d9-c19deb63d6a4-service-ca-bundle\") pod \"insights-operator-555d9d7885-tmwrz\" (UID: \"b92733f1-3721-44de-a7d9-c19deb63d6a4\") " pod="openshift-insights/insights-operator-555d9d7885-tmwrz"
Jul 17 18:37:16.925547 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.925361    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f86765ef-cbaf-4c55-aa42-8451d8e07363-serving-cert\") pod \"service-ca-operator-5bcbc94c78-thxzx\" (UID: \"f86765ef-cbaf-4c55-aa42-8451d8e07363\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-thxzx"
Jul 17 18:37:16.925773 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.925665    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b92733f1-3721-44de-a7d9-c19deb63d6a4-tmp\") pod \"insights-operator-555d9d7885-tmwrz\" (UID: \"b92733f1-3721-44de-a7d9-c19deb63d6a4\") " pod="openshift-insights/insights-operator-555d9d7885-tmwrz"
Jul 17 18:37:16.925941 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.925915    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b92733f1-3721-44de-a7d9-c19deb63d6a4-snapshots\") pod \"insights-operator-555d9d7885-tmwrz\" (UID: \"b92733f1-3721-44de-a7d9-c19deb63d6a4\") " pod="openshift-insights/insights-operator-555d9d7885-tmwrz"
Jul 17 18:37:16.926199 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.926020    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f86765ef-cbaf-4c55-aa42-8451d8e07363-config\") pod \"service-ca-operator-5bcbc94c78-thxzx\" (UID: \"f86765ef-cbaf-4c55-aa42-8451d8e07363\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-thxzx"
Jul 17 18:37:16.926199 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.926075    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b92733f1-3721-44de-a7d9-c19deb63d6a4-service-ca-bundle\") pod \"insights-operator-555d9d7885-tmwrz\" (UID: \"b92733f1-3721-44de-a7d9-c19deb63d6a4\") " pod="openshift-insights/insights-operator-555d9d7885-tmwrz"
Jul 17 18:37:16.926199 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.926154    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b92733f1-3721-44de-a7d9-c19deb63d6a4-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-tmwrz\" (UID: \"b92733f1-3721-44de-a7d9-c19deb63d6a4\") " pod="openshift-insights/insights-operator-555d9d7885-tmwrz"
Jul 17 18:37:16.927665 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.927643    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f86765ef-cbaf-4c55-aa42-8451d8e07363-serving-cert\") pod \"service-ca-operator-5bcbc94c78-thxzx\" (UID: \"f86765ef-cbaf-4c55-aa42-8451d8e07363\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-thxzx"
Jul 17 18:37:16.927892 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.927873    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b92733f1-3721-44de-a7d9-c19deb63d6a4-serving-cert\") pod \"insights-operator-555d9d7885-tmwrz\" (UID: \"b92733f1-3721-44de-a7d9-c19deb63d6a4\") " pod="openshift-insights/insights-operator-555d9d7885-tmwrz"
Jul 17 18:37:16.936435 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.936417    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ttz9\" (UniqueName: \"kubernetes.io/projected/b92733f1-3721-44de-a7d9-c19deb63d6a4-kube-api-access-2ttz9\") pod \"insights-operator-555d9d7885-tmwrz\" (UID: \"b92733f1-3721-44de-a7d9-c19deb63d6a4\") " pod="openshift-insights/insights-operator-555d9d7885-tmwrz"
Jul 17 18:37:16.938497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.938475    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k9d9\" (UniqueName: \"kubernetes.io/projected/f86765ef-cbaf-4c55-aa42-8451d8e07363-kube-api-access-9k9d9\") pod \"service-ca-operator-5bcbc94c78-thxzx\" (UID: \"f86765ef-cbaf-4c55-aa42-8451d8e07363\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-thxzx"
Jul 17 18:37:16.979457 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:16.979426    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-vv4vz"
Jul 17 18:37:17.077208 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:17.077180    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-tmwrz"
Jul 17 18:37:17.082207 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:17.082181    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-thxzx"
Jul 17 18:37:17.104750 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:17.104702    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-vv4vz"]
Jul 17 18:37:17.107921 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:37:17.107877    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5bd5031_5dc3_446b_88ea_83821a9f3faf.slice/crio-09c5049af22b57dbd4b1c3980876165bf35659052613f5f93bc00ae74e17002d WatchSource:0}: Error finding container 09c5049af22b57dbd4b1c3980876165bf35659052613f5f93bc00ae74e17002d: Status 404 returned error can't find the container with id 09c5049af22b57dbd4b1c3980876165bf35659052613f5f93bc00ae74e17002d
Jul 17 18:37:17.214173 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:17.214150    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-tmwrz"]
Jul 17 18:37:17.216390 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:37:17.216364    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92733f1_3721_44de_a7d9_c19deb63d6a4.slice/crio-7c51a5a802ce8e77cad023361fef49af593e41ed9cffdfacb154e02cd41f90b8 WatchSource:0}: Error finding container 7c51a5a802ce8e77cad023361fef49af593e41ed9cffdfacb154e02cd41f90b8: Status 404 returned error can't find the container with id 7c51a5a802ce8e77cad023361fef49af593e41ed9cffdfacb154e02cd41f90b8
Jul 17 18:37:17.237676 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:17.237646    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-thxzx"]
Jul 17 18:37:17.241343 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:37:17.241291    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86765ef_cbaf_4c55_aa42_8451d8e07363.slice/crio-2202fdf9b00cd9cde291fc37425c927fc4d24dd17553c5cc8148b5e129ddea61 WatchSource:0}: Error finding container 2202fdf9b00cd9cde291fc37425c927fc4d24dd17553c5cc8148b5e129ddea61: Status 404 returned error can't find the container with id 2202fdf9b00cd9cde291fc37425c927fc4d24dd17553c5cc8148b5e129ddea61
Jul 17 18:37:17.327978 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:17.327944    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-xhz5l\" (UID: \"39a38b78-ab33-411b-9a10-4cd932d6c535\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l"
Jul 17 18:37:17.328144 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:17.327998    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c468674e-6f4b-4a05-abf6-87624054ac4c-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ks5dn\" (UID: \"c468674e-6f4b-4a05-abf6-87624054ac4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks5dn"
Jul 17 18:37:17.328144 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:17.328081    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 18:37:17.328209 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:17.328149    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls podName:39a38b78-ab33-411b-9a10-4cd932d6c535 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:18.328134326 +0000 UTC m=+248.769642993 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-xhz5l" (UID: "39a38b78-ab33-411b-9a10-4cd932d6c535") : secret "cluster-monitoring-operator-tls" not found
Jul 17 18:37:17.328209 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:17.328085    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 18:37:17.328275 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:17.328219    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c468674e-6f4b-4a05-abf6-87624054ac4c-samples-operator-tls podName:c468674e-6f4b-4a05-abf6-87624054ac4c nodeName:}" failed. No retries permitted until 2026-07-17 18:37:18.328206929 +0000 UTC m=+248.769715596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c468674e-6f4b-4a05-abf6-87624054ac4c-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ks5dn" (UID: "c468674e-6f4b-4a05-abf6-87624054ac4c") : secret "samples-operator-tls" not found
Jul 17 18:37:17.872977 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:17.872873    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-thxzx" event={"ID":"f86765ef-cbaf-4c55-aa42-8451d8e07363","Type":"ContainerStarted","Data":"2202fdf9b00cd9cde291fc37425c927fc4d24dd17553c5cc8148b5e129ddea61"}
Jul 17 18:37:17.874809 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:17.874755    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-vv4vz" event={"ID":"b5bd5031-5dc3-446b-88ea-83821a9f3faf","Type":"ContainerStarted","Data":"09c5049af22b57dbd4b1c3980876165bf35659052613f5f93bc00ae74e17002d"}
Jul 17 18:37:17.876160 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:17.876133    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-tmwrz" event={"ID":"b92733f1-3721-44de-a7d9-c19deb63d6a4","Type":"ContainerStarted","Data":"7c51a5a802ce8e77cad023361fef49af593e41ed9cffdfacb154e02cd41f90b8"}
Jul 17 18:37:18.336343 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:18.336095    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c468674e-6f4b-4a05-abf6-87624054ac4c-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ks5dn\" (UID: \"c468674e-6f4b-4a05-abf6-87624054ac4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks5dn"
Jul 17 18:37:18.336343 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:18.336189    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-xhz5l\" (UID: \"39a38b78-ab33-411b-9a10-4cd932d6c535\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l"
Jul 17 18:37:18.336343 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:18.336295    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 18:37:18.336343 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:18.336292    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 18:37:18.336712 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:18.336380    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls podName:39a38b78-ab33-411b-9a10-4cd932d6c535 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:20.336360895 +0000 UTC m=+250.777869562 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-xhz5l" (UID: "39a38b78-ab33-411b-9a10-4cd932d6c535") : secret "cluster-monitoring-operator-tls" not found
Jul 17 18:37:18.336712 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:18.336398    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c468674e-6f4b-4a05-abf6-87624054ac4c-samples-operator-tls podName:c468674e-6f4b-4a05-abf6-87624054ac4c nodeName:}" failed. No retries permitted until 2026-07-17 18:37:20.33638945 +0000 UTC m=+250.777898117 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c468674e-6f4b-4a05-abf6-87624054ac4c-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ks5dn" (UID: "c468674e-6f4b-4a05-abf6-87624054ac4c") : secret "samples-operator-tls" not found
Jul 17 18:37:20.355556 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:20.355526    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c468674e-6f4b-4a05-abf6-87624054ac4c-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ks5dn\" (UID: \"c468674e-6f4b-4a05-abf6-87624054ac4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks5dn"
Jul 17 18:37:20.355857 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:20.355592    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-xhz5l\" (UID: \"39a38b78-ab33-411b-9a10-4cd932d6c535\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l"
Jul 17 18:37:20.355857 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:20.355697    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 18:37:20.355857 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:20.355744    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls podName:39a38b78-ab33-411b-9a10-4cd932d6c535 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:24.355730022 +0000 UTC m=+254.797238688 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-xhz5l" (UID: "39a38b78-ab33-411b-9a10-4cd932d6c535") : secret "cluster-monitoring-operator-tls" not found
Jul 17 18:37:20.356051 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:20.356036    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 18:37:20.356085 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:20.356074    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c468674e-6f4b-4a05-abf6-87624054ac4c-samples-operator-tls podName:c468674e-6f4b-4a05-abf6-87624054ac4c nodeName:}" failed. No retries permitted until 2026-07-17 18:37:24.356064693 +0000 UTC m=+254.797573360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c468674e-6f4b-4a05-abf6-87624054ac4c-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ks5dn" (UID: "c468674e-6f4b-4a05-abf6-87624054ac4c") : secret "samples-operator-tls" not found
Jul 17 18:37:20.889355 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:20.889297    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-tmwrz" event={"ID":"b92733f1-3721-44de-a7d9-c19deb63d6a4","Type":"ContainerStarted","Data":"9e799d84400c0bfb99b3301b4cde83d2a9492fc451d7bf7d982d31dd5f0cfe70"}
Jul 17 18:37:20.890748 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:20.890715    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-thxzx" event={"ID":"f86765ef-cbaf-4c55-aa42-8451d8e07363","Type":"ContainerStarted","Data":"b52d3acaa16863dcc314b217f6e21780f738dcfb575ada2891419112f6dd3a5c"}
Jul 17 18:37:20.892144 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:20.892126    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/0.log"
Jul 17 18:37:20.892224 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:20.892162    2559 generic.go:356] "Generic (PLEG): container finished" podID="b5bd5031-5dc3-446b-88ea-83821a9f3faf" containerID="b2b059ce13424884f21c53fb9c4fff8d44452069072d7eff83886bdbbadfbc07" exitCode=255
Jul 17 18:37:20.892224 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:20.892187    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-vv4vz" event={"ID":"b5bd5031-5dc3-446b-88ea-83821a9f3faf","Type":"ContainerDied","Data":"b2b059ce13424884f21c53fb9c4fff8d44452069072d7eff83886bdbbadfbc07"}
Jul 17 18:37:20.892378 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:20.892365    2559 scope.go:117] "RemoveContainer" containerID="b2b059ce13424884f21c53fb9c4fff8d44452069072d7eff83886bdbbadfbc07"
Jul 17 18:37:20.908023 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:20.907963    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-tmwrz" podStartSLOduration=1.913105876 podStartE2EDuration="4.907946624s" totalImagesPullingTime="2.994840748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:20.907918245 +0000 UTC m=+251.349426937" watchObservedRunningTime="2026-07-17 18:37:20.907946624 +0000 UTC m=+251.349455314"
Jul 17 18:37:20.928124 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:20.928071    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-thxzx" podStartSLOduration=1.959106315 podStartE2EDuration="4.928051944s" totalImagesPullingTime="2.968945629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:20.926967951 +0000 UTC m=+251.368476641" watchObservedRunningTime="2026-07-17 18:37:20.928051944 +0000 UTC m=+251.369560628"
Jul 17 18:37:21.077261 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.077227    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwbpb"]
Jul 17 18:37:21.079360 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.079343    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwbpb"
Jul 17 18:37:21.082086 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.082054    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 17 18:37:21.082697 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.082679    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-d4c8n\""
Jul 17 18:37:21.082954 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.082933    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 17 18:37:21.099265 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.099239    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwbpb"]
Jul 17 18:37:21.161722 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.161646    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9dg5\" (UniqueName: \"kubernetes.io/projected/03b5f935-6b84-4179-93f1-86e325c90c81-kube-api-access-n9dg5\") pod \"migrator-7cc9c7ffd7-cwbpb\" (UID: \"03b5f935-6b84-4179-93f1-86e325c90c81\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwbpb"
Jul 17 18:37:21.266959 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.262939    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9dg5\" (UniqueName: \"kubernetes.io/projected/03b5f935-6b84-4179-93f1-86e325c90c81-kube-api-access-n9dg5\") pod \"migrator-7cc9c7ffd7-cwbpb\" (UID: \"03b5f935-6b84-4179-93f1-86e325c90c81\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwbpb"
Jul 17 18:37:21.273545 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.273524    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9dg5\" (UniqueName: \"kubernetes.io/projected/03b5f935-6b84-4179-93f1-86e325c90c81-kube-api-access-n9dg5\") pod \"migrator-7cc9c7ffd7-cwbpb\" (UID: \"03b5f935-6b84-4179-93f1-86e325c90c81\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwbpb"
Jul 17 18:37:21.388621 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.388586    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwbpb"
Jul 17 18:37:21.401357 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.401325    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bzs5r"]
Jul 17 18:37:21.404171 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.404147    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:21.408696 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.408099    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jvlrf\""
Jul 17 18:37:21.408696 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.408327    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 17 18:37:21.408696 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.408528    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 17 18:37:21.422240 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.422210    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bzs5r"]
Jul 17 18:37:21.464662 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.464370    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a85d465e-b514-4d92-9869-898394cd3652-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:21.464662 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.464449    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a85d465e-b514-4d92-9869-898394cd3652-data-volume\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:21.464662 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.464510    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a85d465e-b514-4d92-9869-898394cd3652-crio-socket\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:21.464662 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.464573    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgsfp\" (UniqueName: \"kubernetes.io/projected/a85d465e-b514-4d92-9869-898394cd3652-kube-api-access-qgsfp\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:21.464662 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.464653    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a85d465e-b514-4d92-9869-898394cd3652-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:21.535658 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.535626    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwbpb"]
Jul 17 18:37:21.538996 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:37:21.538968    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b5f935_6b84_4179_93f1_86e325c90c81.slice/crio-d6118a1ef1a78477e4072c870c739df268498d6284f6607ec8435cd69fddc579 WatchSource:0}: Error finding container d6118a1ef1a78477e4072c870c739df268498d6284f6607ec8435cd69fddc579: Status 404 returned error can't find the container with id d6118a1ef1a78477e4072c870c739df268498d6284f6607ec8435cd69fddc579
Jul 17 18:37:21.565427 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.565397    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a85d465e-b514-4d92-9869-898394cd3652-data-volume\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:21.565528 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.565446    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a85d465e-b514-4d92-9869-898394cd3652-crio-socket\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:21.565592 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.565580    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a85d465e-b514-4d92-9869-898394cd3652-crio-socket\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:21.565637 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.565625    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgsfp\" (UniqueName: \"kubernetes.io/projected/a85d465e-b514-4d92-9869-898394cd3652-kube-api-access-qgsfp\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:21.565679 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.565651    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a85d465e-b514-4d92-9869-898394cd3652-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:21.565714 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.565688    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a85d465e-b514-4d92-9869-898394cd3652-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:21.565795 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.565772    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a85d465e-b514-4d92-9869-898394cd3652-data-volume\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:21.565828 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:21.565784    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 18:37:21.565873 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:21.565787    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 18:37:21.565908 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:21.565873    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a85d465e-b514-4d92-9869-898394cd3652-kube-rbac-proxy-cm podName:a85d465e-b514-4d92-9869-898394cd3652 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:22.065852579 +0000 UTC m=+252.507361251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a85d465e-b514-4d92-9869-898394cd3652-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzs5r" (UID: "a85d465e-b514-4d92-9869-898394cd3652") : configmap "kube-rbac-proxy" not found
Jul 17 18:37:21.565908 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:21.565900    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a85d465e-b514-4d92-9869-898394cd3652-insights-runtime-extractor-tls podName:a85d465e-b514-4d92-9869-898394cd3652 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:22.065889582 +0000 UTC m=+252.507398249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a85d465e-b514-4d92-9869-898394cd3652-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzs5r" (UID: "a85d465e-b514-4d92-9869-898394cd3652") : secret "insights-runtime-extractor-tls" not found
Jul 17 18:37:21.575617 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.575595    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgsfp\" (UniqueName: \"kubernetes.io/projected/a85d465e-b514-4d92-9869-898394cd3652-kube-api-access-qgsfp\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:21.767434 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.767354    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs\") pod \"network-metrics-daemon-tt279\" (UID: \"f9fc2de6-adf3-45c0-952a-e317f362aa51\") " pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:37:21.767565 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:21.767510    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 17 18:37:21.767609 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:21.767580    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs podName:f9fc2de6-adf3-45c0-952a-e317f362aa51 nodeName:}" failed. No retries permitted until 2026-07-17 18:39:23.767565958 +0000 UTC m=+374.209074625 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs") pod "network-metrics-daemon-tt279" (UID: "f9fc2de6-adf3-45c0-952a-e317f362aa51") : secret "metrics-daemon-secret" not found
Jul 17 18:37:21.896394 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.896369    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 18:37:21.896867 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.896834    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/0.log"
Jul 17 18:37:21.897009 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.896883    2559 generic.go:356] "Generic (PLEG): container finished" podID="b5bd5031-5dc3-446b-88ea-83821a9f3faf" containerID="02b304ad6f1f4ba028d7c331e57fb0d578a7882e57107dd02194f0d2e493bf2e" exitCode=255
Jul 17 18:37:21.897009 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.896959    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-vv4vz" event={"ID":"b5bd5031-5dc3-446b-88ea-83821a9f3faf","Type":"ContainerDied","Data":"02b304ad6f1f4ba028d7c331e57fb0d578a7882e57107dd02194f0d2e493bf2e"}
Jul 17 18:37:21.897125 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.897024    2559 scope.go:117] "RemoveContainer" containerID="b2b059ce13424884f21c53fb9c4fff8d44452069072d7eff83886bdbbadfbc07"
Jul 17 18:37:21.897247 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.897229    2559 scope.go:117] "RemoveContainer" containerID="02b304ad6f1f4ba028d7c331e57fb0d578a7882e57107dd02194f0d2e493bf2e"
Jul 17 18:37:21.897480 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:21.897454    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-vv4vz_openshift-console-operator(b5bd5031-5dc3-446b-88ea-83821a9f3faf)\"" pod="openshift-console-operator/console-operator-565b7b569d-vv4vz" podUID="b5bd5031-5dc3-446b-88ea-83821a9f3faf"
Jul 17 18:37:21.898450 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:21.898427    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwbpb" event={"ID":"03b5f935-6b84-4179-93f1-86e325c90c81","Type":"ContainerStarted","Data":"d6118a1ef1a78477e4072c870c739df268498d6284f6607ec8435cd69fddc579"}
Jul 17 18:37:22.070284 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:22.070249    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a85d465e-b514-4d92-9869-898394cd3652-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:22.070533 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:22.070294    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a85d465e-b514-4d92-9869-898394cd3652-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:22.070533 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:22.070409    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 18:37:22.070533 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:22.070429    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 18:37:22.070533 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:22.070491    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a85d465e-b514-4d92-9869-898394cd3652-kube-rbac-proxy-cm podName:a85d465e-b514-4d92-9869-898394cd3652 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:23.07046602 +0000 UTC m=+253.511974706 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a85d465e-b514-4d92-9869-898394cd3652-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzs5r" (UID: "a85d465e-b514-4d92-9869-898394cd3652") : configmap "kube-rbac-proxy" not found
Jul 17 18:37:22.070533 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:22.070513    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a85d465e-b514-4d92-9869-898394cd3652-insights-runtime-extractor-tls podName:a85d465e-b514-4d92-9869-898394cd3652 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:23.070502137 +0000 UTC m=+253.512010810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a85d465e-b514-4d92-9869-898394cd3652-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzs5r" (UID: "a85d465e-b514-4d92-9869-898394cd3652") : secret "insights-runtime-extractor-tls" not found
Jul 17 18:37:22.905645 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:22.905572    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 18:37:22.906073 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:22.905933    2559 scope.go:117] "RemoveContainer" containerID="02b304ad6f1f4ba028d7c331e57fb0d578a7882e57107dd02194f0d2e493bf2e"
Jul 17 18:37:22.906123 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:22.906098    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-vv4vz_openshift-console-operator(b5bd5031-5dc3-446b-88ea-83821a9f3faf)\"" pod="openshift-console-operator/console-operator-565b7b569d-vv4vz" podUID="b5bd5031-5dc3-446b-88ea-83821a9f3faf"
Jul 17 18:37:22.907274 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:22.907252    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwbpb" event={"ID":"03b5f935-6b84-4179-93f1-86e325c90c81","Type":"ContainerStarted","Data":"3e7ed631b04b223c80fe9bfaeca019041f732adea7e631f2e29d3ed488c301f2"}
Jul 17 18:37:22.907400 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:22.907293    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwbpb" event={"ID":"03b5f935-6b84-4179-93f1-86e325c90c81","Type":"ContainerStarted","Data":"48126bded47fb29cee42be9fc9b7f1e38becd4700c735913b1d3671e464f61f8"}
Jul 17 18:37:23.079842 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:23.079784    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a85d465e-b514-4d92-9869-898394cd3652-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:23.080029 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:23.079867    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a85d465e-b514-4d92-9869-898394cd3652-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:23.080029 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:23.079909    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 18:37:23.080029 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:23.079977    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a85d465e-b514-4d92-9869-898394cd3652-kube-rbac-proxy-cm podName:a85d465e-b514-4d92-9869-898394cd3652 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:25.079957155 +0000 UTC m=+255.521465822 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a85d465e-b514-4d92-9869-898394cd3652-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzs5r" (UID: "a85d465e-b514-4d92-9869-898394cd3652") : configmap "kube-rbac-proxy" not found
Jul 17 18:37:23.080029 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:23.080000    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 18:37:23.080170 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:23.080055    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a85d465e-b514-4d92-9869-898394cd3652-insights-runtime-extractor-tls podName:a85d465e-b514-4d92-9869-898394cd3652 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:25.080040318 +0000 UTC m=+255.521548984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a85d465e-b514-4d92-9869-898394cd3652-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzs5r" (UID: "a85d465e-b514-4d92-9869-898394cd3652") : secret "insights-runtime-extractor-tls" not found
Jul 17 18:37:24.194178 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.194148    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-njrqr_3a1608b8-a53a-480d-965d-8ab7e3ec8f22/dns-node-resolver/0.log"
Jul 17 18:37:24.391259 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.391222    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-xhz5l\" (UID: \"39a38b78-ab33-411b-9a10-4cd932d6c535\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l"
Jul 17 18:37:24.391466 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.391277    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c468674e-6f4b-4a05-abf6-87624054ac4c-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ks5dn\" (UID: \"c468674e-6f4b-4a05-abf6-87624054ac4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks5dn"
Jul 17 18:37:24.391466 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:24.391423    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 18:37:24.391466 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:24.391453    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 18:37:24.391622 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:24.391493    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls podName:39a38b78-ab33-411b-9a10-4cd932d6c535 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:32.391473573 +0000 UTC m=+262.832982257 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-xhz5l" (UID: "39a38b78-ab33-411b-9a10-4cd932d6c535") : secret "cluster-monitoring-operator-tls" not found
Jul 17 18:37:24.391622 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:24.391515    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c468674e-6f4b-4a05-abf6-87624054ac4c-samples-operator-tls podName:c468674e-6f4b-4a05-abf6-87624054ac4c nodeName:}" failed. No retries permitted until 2026-07-17 18:37:32.391502913 +0000 UTC m=+262.833011580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c468674e-6f4b-4a05-abf6-87624054ac4c-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ks5dn" (UID: "c468674e-6f4b-4a05-abf6-87624054ac4c") : secret "samples-operator-tls" not found
Jul 17 18:37:24.424364 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.424297    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwbpb" podStartSLOduration=2.4032471539999998 podStartE2EDuration="3.424279206s" totalImagesPullingTime="1.021032052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:22.948635532 +0000 UTC m=+253.390144222" watchObservedRunningTime="2026-07-17 18:37:24.424279206 +0000 UTC m=+254.865787962"
Jul 17 18:37:24.424813 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.424799    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-k4cwz"]
Jul 17 18:37:24.426888 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.426875    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-k4cwz"
Jul 17 18:37:24.431233 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.431215    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 17 18:37:24.432436 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.432419    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 17 18:37:24.432519 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.432435    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 17 18:37:24.432519 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.432418    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5wc9s\""
Jul 17 18:37:24.433480 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.433336    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 17 18:37:24.444477 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.444429    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-k4cwz"]
Jul 17 18:37:24.492519 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.492484    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qgnm\" (UniqueName: \"kubernetes.io/projected/a88fde0a-76cd-4707-b335-0beb8fac0cd8-kube-api-access-4qgnm\") pod \"service-ca-58598695f7-k4cwz\" (UID: \"a88fde0a-76cd-4707-b335-0beb8fac0cd8\") " pod="openshift-service-ca/service-ca-58598695f7-k4cwz"
Jul 17 18:37:24.492688 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.492549    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a88fde0a-76cd-4707-b335-0beb8fac0cd8-signing-cabundle\") pod \"service-ca-58598695f7-k4cwz\" (UID: \"a88fde0a-76cd-4707-b335-0beb8fac0cd8\") " pod="openshift-service-ca/service-ca-58598695f7-k4cwz"
Jul 17 18:37:24.492688 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.492590    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a88fde0a-76cd-4707-b335-0beb8fac0cd8-signing-key\") pod \"service-ca-58598695f7-k4cwz\" (UID: \"a88fde0a-76cd-4707-b335-0beb8fac0cd8\") " pod="openshift-service-ca/service-ca-58598695f7-k4cwz"
Jul 17 18:37:24.592964 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.592928    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qgnm\" (UniqueName: \"kubernetes.io/projected/a88fde0a-76cd-4707-b335-0beb8fac0cd8-kube-api-access-4qgnm\") pod \"service-ca-58598695f7-k4cwz\" (UID: \"a88fde0a-76cd-4707-b335-0beb8fac0cd8\") " pod="openshift-service-ca/service-ca-58598695f7-k4cwz"
Jul 17 18:37:24.593164 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.592994    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a88fde0a-76cd-4707-b335-0beb8fac0cd8-signing-cabundle\") pod \"service-ca-58598695f7-k4cwz\" (UID: \"a88fde0a-76cd-4707-b335-0beb8fac0cd8\") " pod="openshift-service-ca/service-ca-58598695f7-k4cwz"
Jul 17 18:37:24.593164 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.593023    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a88fde0a-76cd-4707-b335-0beb8fac0cd8-signing-key\") pod \"service-ca-58598695f7-k4cwz\" (UID: \"a88fde0a-76cd-4707-b335-0beb8fac0cd8\") " pod="openshift-service-ca/service-ca-58598695f7-k4cwz"
Jul 17 18:37:24.593719 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.593696    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a88fde0a-76cd-4707-b335-0beb8fac0cd8-signing-cabundle\") pod \"service-ca-58598695f7-k4cwz\" (UID: \"a88fde0a-76cd-4707-b335-0beb8fac0cd8\") " pod="openshift-service-ca/service-ca-58598695f7-k4cwz"
Jul 17 18:37:24.595659 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.595637    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a88fde0a-76cd-4707-b335-0beb8fac0cd8-signing-key\") pod \"service-ca-58598695f7-k4cwz\" (UID: \"a88fde0a-76cd-4707-b335-0beb8fac0cd8\") " pod="openshift-service-ca/service-ca-58598695f7-k4cwz"
Jul 17 18:37:24.605838 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.605814    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qgnm\" (UniqueName: \"kubernetes.io/projected/a88fde0a-76cd-4707-b335-0beb8fac0cd8-kube-api-access-4qgnm\") pod \"service-ca-58598695f7-k4cwz\" (UID: \"a88fde0a-76cd-4707-b335-0beb8fac0cd8\") " pod="openshift-service-ca/service-ca-58598695f7-k4cwz"
Jul 17 18:37:24.735929 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.735845    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-k4cwz"
Jul 17 18:37:24.866633 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.866595    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-k4cwz"]
Jul 17 18:37:24.869590 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:37:24.869565    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88fde0a_76cd_4707_b335_0beb8fac0cd8.slice/crio-0d343581e6908597d490dbfd6456ea3cc14353b7611624f55a17150fbbd762d9 WatchSource:0}: Error finding container 0d343581e6908597d490dbfd6456ea3cc14353b7611624f55a17150fbbd762d9: Status 404 returned error can't find the container with id 0d343581e6908597d490dbfd6456ea3cc14353b7611624f55a17150fbbd762d9
Jul 17 18:37:24.915223 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:24.915197    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-k4cwz" event={"ID":"a88fde0a-76cd-4707-b335-0beb8fac0cd8","Type":"ContainerStarted","Data":"0d343581e6908597d490dbfd6456ea3cc14353b7611624f55a17150fbbd762d9"}
Jul 17 18:37:25.097411 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:25.097380    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a85d465e-b514-4d92-9869-898394cd3652-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:25.097585 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:25.097430    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a85d465e-b514-4d92-9869-898394cd3652-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:25.097585 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:25.097518    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 18:37:25.097585 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:25.097549    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 18:37:25.097699 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:25.097597    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a85d465e-b514-4d92-9869-898394cd3652-kube-rbac-proxy-cm podName:a85d465e-b514-4d92-9869-898394cd3652 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:29.097576171 +0000 UTC m=+259.539084851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a85d465e-b514-4d92-9869-898394cd3652-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzs5r" (UID: "a85d465e-b514-4d92-9869-898394cd3652") : configmap "kube-rbac-proxy" not found
Jul 17 18:37:25.097699 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:25.097616    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a85d465e-b514-4d92-9869-898394cd3652-insights-runtime-extractor-tls podName:a85d465e-b514-4d92-9869-898394cd3652 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:29.097607345 +0000 UTC m=+259.539116017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a85d465e-b514-4d92-9869-898394cd3652-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzs5r" (UID: "a85d465e-b514-4d92-9869-898394cd3652") : secret "insights-runtime-extractor-tls" not found
Jul 17 18:37:25.399788 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:25.399718    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ztfrm_d33d089a-3e2f-4edd-a459-0303cf90f1af/node-ca/0.log"
Jul 17 18:37:25.920408 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:25.920369    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-k4cwz" event={"ID":"a88fde0a-76cd-4707-b335-0beb8fac0cd8","Type":"ContainerStarted","Data":"554ec5db922519256b35c1709d784dbc6efd340a35203aec591518ae1f267837"}
Jul 17 18:37:25.938601 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:25.938554    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-k4cwz" podStartSLOduration=1.9385406710000002 podStartE2EDuration="1.938540671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:25.938027887 +0000 UTC m=+256.379536575" watchObservedRunningTime="2026-07-17 18:37:25.938540671 +0000 UTC m=+256.380049375"
Jul 17 18:37:26.198159 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:26.198077    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lfvbt_0881b192-5909-49a5-ba16-65dbc6829189/serve-healthcheck-canary/0.log"
Jul 17 18:37:26.395024 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:26.394996    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-cwbpb_03b5f935-6b84-4179-93f1-86e325c90c81/migrator/0.log"
Jul 17 18:37:26.592506 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:26.592472    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-cwbpb_03b5f935-6b84-4179-93f1-86e325c90c81/graceful-termination/0.log"
Jul 17 18:37:26.980646 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:26.980551    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-vv4vz"
Jul 17 18:37:26.980646 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:26.980623    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-vv4vz"
Jul 17 18:37:26.981060 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:26.981047    2559 scope.go:117] "RemoveContainer" containerID="02b304ad6f1f4ba028d7c331e57fb0d578a7882e57107dd02194f0d2e493bf2e"
Jul 17 18:37:26.981251 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:26.981230    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-vv4vz_openshift-console-operator(b5bd5031-5dc3-446b-88ea-83821a9f3faf)\"" pod="openshift-console-operator/console-operator-565b7b569d-vv4vz" podUID="b5bd5031-5dc3-446b-88ea-83821a9f3faf"
Jul 17 18:37:29.131199 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:29.131157    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a85d465e-b514-4d92-9869-898394cd3652-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:29.131687 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:29.131208    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a85d465e-b514-4d92-9869-898394cd3652-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:29.131687 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:29.131283    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 18:37:29.131687 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:29.131362    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a85d465e-b514-4d92-9869-898394cd3652-kube-rbac-proxy-cm podName:a85d465e-b514-4d92-9869-898394cd3652 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:37.131345389 +0000 UTC m=+267.572854057 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a85d465e-b514-4d92-9869-898394cd3652-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bzs5r" (UID: "a85d465e-b514-4d92-9869-898394cd3652") : configmap "kube-rbac-proxy" not found
Jul 17 18:37:29.131687 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:29.131376    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 18:37:29.131687 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:29.131423    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a85d465e-b514-4d92-9869-898394cd3652-insights-runtime-extractor-tls podName:a85d465e-b514-4d92-9869-898394cd3652 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:37.13141052 +0000 UTC m=+267.572919187 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a85d465e-b514-4d92-9869-898394cd3652-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bzs5r" (UID: "a85d465e-b514-4d92-9869-898394cd3652") : secret "insights-runtime-extractor-tls" not found
Jul 17 18:37:32.457383 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:32.457284    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c468674e-6f4b-4a05-abf6-87624054ac4c-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ks5dn\" (UID: \"c468674e-6f4b-4a05-abf6-87624054ac4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks5dn"
Jul 17 18:37:32.457838 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:32.457406    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-xhz5l\" (UID: \"39a38b78-ab33-411b-9a10-4cd932d6c535\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l"
Jul 17 18:37:32.457838 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:32.457504    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 18:37:32.457838 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:32.457567    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls podName:39a38b78-ab33-411b-9a10-4cd932d6c535 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:48.457548882 +0000 UTC m=+278.899057548 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-xhz5l" (UID: "39a38b78-ab33-411b-9a10-4cd932d6c535") : secret "cluster-monitoring-operator-tls" not found
Jul 17 18:37:32.459940 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:32.459914    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c468674e-6f4b-4a05-abf6-87624054ac4c-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ks5dn\" (UID: \"c468674e-6f4b-4a05-abf6-87624054ac4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks5dn"
Jul 17 18:37:32.591720 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:32.591692    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks5dn"
Jul 17 18:37:32.719487 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:32.719404    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks5dn"]
Jul 17 18:37:32.942359 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:32.942320    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks5dn" event={"ID":"c468674e-6f4b-4a05-abf6-87624054ac4c","Type":"ContainerStarted","Data":"1425b8584802f116e9a27a170f79512aab10f8be31fb6e174f3a14bafe2d6788"}
Jul 17 18:37:35.955298 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:35.955262    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks5dn" event={"ID":"c468674e-6f4b-4a05-abf6-87624054ac4c","Type":"ContainerStarted","Data":"c948798466d62965a683436f78cd76dba25e937d53ac5cfa471ddccba84a9a75"}
Jul 17 18:37:35.955753 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:35.955328    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks5dn" event={"ID":"c468674e-6f4b-4a05-abf6-87624054ac4c","Type":"ContainerStarted","Data":"7bc379be12afbbb619361deaccb29e2ee310f82a1dcc0ad19580f7fa4a4e4bd0"}
Jul 17 18:37:35.976526 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:35.976479    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ks5dn" podStartSLOduration=17.278708985 podStartE2EDuration="19.97646279s" totalImagesPullingTime="2.697753805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:35.975774654 +0000 UTC m=+266.417283341" watchObservedRunningTime="2026-07-17 18:37:35.97646279 +0000 UTC m=+266.417971499"
Jul 17 18:37:37.197797 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:37.197760    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a85d465e-b514-4d92-9869-898394cd3652-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:37.197797 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:37.197807    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a85d465e-b514-4d92-9869-898394cd3652-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:37.198367 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:37.198347    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a85d465e-b514-4d92-9869-898394cd3652-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:37.200432 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:37.200411    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a85d465e-b514-4d92-9869-898394cd3652-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzs5r\" (UID: \"a85d465e-b514-4d92-9869-898394cd3652\") " pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:37.320027 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:37.319989    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bzs5r"
Jul 17 18:37:37.452200 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:37.452178    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bzs5r"]
Jul 17 18:37:37.454743 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:37:37.454718    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85d465e_b514_4d92_9869_898394cd3652.slice/crio-ce88dc2f4871ff6a352eb1d9e9e1955afd53e4d475ab1a451091e37e6b867ce4 WatchSource:0}: Error finding container ce88dc2f4871ff6a352eb1d9e9e1955afd53e4d475ab1a451091e37e6b867ce4: Status 404 returned error can't find the container with id ce88dc2f4871ff6a352eb1d9e9e1955afd53e4d475ab1a451091e37e6b867ce4
Jul 17 18:37:37.966844 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:37.966813    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzs5r" event={"ID":"a85d465e-b514-4d92-9869-898394cd3652","Type":"ContainerStarted","Data":"ce88dc2f4871ff6a352eb1d9e9e1955afd53e4d475ab1a451091e37e6b867ce4"}
Jul 17 18:37:38.125143 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:38.124978    2559 scope.go:117] "RemoveContainer" containerID="02b304ad6f1f4ba028d7c331e57fb0d578a7882e57107dd02194f0d2e493bf2e"
Jul 17 18:37:38.972598 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:38.972567    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 18:37:38.973126 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:38.972652    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-vv4vz" event={"ID":"b5bd5031-5dc3-446b-88ea-83821a9f3faf","Type":"ContainerStarted","Data":"aa2eb4494663fedb8d0c1c4acff86a323ddc547effa1008a7a78625b331296d1"}
Jul 17 18:37:38.973126 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:38.972930    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-vv4vz"
Jul 17 18:37:38.997429 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:38.997372    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-vv4vz" podStartSLOduration=19.899222901 podStartE2EDuration="22.997351454s" totalImagesPullingTime="3.098128553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:38.995657919 +0000 UTC m=+269.437166673" watchObservedRunningTime="2026-07-17 18:37:38.997351454 +0000 UTC m=+269.438860144"
Jul 17 18:37:39.134875 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:39.134840    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-vv4vz"
Jul 17 18:37:42.989426 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:42.989390    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzs5r" event={"ID":"a85d465e-b514-4d92-9869-898394cd3652","Type":"ContainerStarted","Data":"96b0e62771faf05dc59cfe7c4e10a406ba252b41180f0849022e91cf5924fdea"}
Jul 17 18:37:46.093631 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.093598    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-549b96bcc9-nl9r2"]
Jul 17 18:37:46.096752 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.096736    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:46.105092 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.105070    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 17 18:37:46.105208 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.105133    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 17 18:37:46.106428 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.106399    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 17 18:37:46.106428 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.106418    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 17 18:37:46.106637 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.106419    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 17 18:37:46.106637 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.106456    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 17 18:37:46.106637 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.106499    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nt4s4\""
Jul 17 18:37:46.111350 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.111334    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 17 18:37:46.115830 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.115810    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-stqc8"]
Jul 17 18:37:46.118819 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.118803    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-stqc8"
Jul 17 18:37:46.121972 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.121953    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-kf6lf\""
Jul 17 18:37:46.136987 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.136963    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-549b96bcc9-nl9r2"]
Jul 17 18:37:46.150620 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.150593    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-stqc8"]
Jul 17 18:37:46.178188 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.178164    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7816aa59-ad03-45df-9901-d7e0dcad2d8c-console-oauth-config\") pod \"console-549b96bcc9-nl9r2\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") " pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:46.178188 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.178191    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7816aa59-ad03-45df-9901-d7e0dcad2d8c-oauth-serving-cert\") pod \"console-549b96bcc9-nl9r2\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") " pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:46.178405 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.178224    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7816aa59-ad03-45df-9901-d7e0dcad2d8c-service-ca\") pod \"console-549b96bcc9-nl9r2\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") " pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:46.178405 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.178344    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-722q2\" (UniqueName: \"kubernetes.io/projected/7816aa59-ad03-45df-9901-d7e0dcad2d8c-kube-api-access-722q2\") pod \"console-549b96bcc9-nl9r2\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") " pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:46.178405 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.178388    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7816aa59-ad03-45df-9901-d7e0dcad2d8c-console-serving-cert\") pod \"console-549b96bcc9-nl9r2\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") " pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:46.178515 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.178410    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7816aa59-ad03-45df-9901-d7e0dcad2d8c-console-config\") pod \"console-549b96bcc9-nl9r2\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") " pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:46.278856 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.278766    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-722q2\" (UniqueName: \"kubernetes.io/projected/7816aa59-ad03-45df-9901-d7e0dcad2d8c-kube-api-access-722q2\") pod \"console-549b96bcc9-nl9r2\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") " pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:46.278856 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.278826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7816aa59-ad03-45df-9901-d7e0dcad2d8c-console-serving-cert\") pod \"console-549b96bcc9-nl9r2\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") " pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:46.279074 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.278905    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7816aa59-ad03-45df-9901-d7e0dcad2d8c-console-config\") pod \"console-549b96bcc9-nl9r2\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") " pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:46.279074 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.278992    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n27qs\" (UniqueName: \"kubernetes.io/projected/7a1c2e51-24a8-49a2-ae10-75ce216f38f8-kube-api-access-n27qs\") pod \"downloads-5bc67dcd46-stqc8\" (UID: \"7a1c2e51-24a8-49a2-ae10-75ce216f38f8\") " pod="openshift-console/downloads-5bc67dcd46-stqc8"
Jul 17 18:37:46.279074 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.279052    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7816aa59-ad03-45df-9901-d7e0dcad2d8c-console-oauth-config\") pod \"console-549b96bcc9-nl9r2\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") " pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:46.279243 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.279114    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7816aa59-ad03-45df-9901-d7e0dcad2d8c-oauth-serving-cert\") pod \"console-549b96bcc9-nl9r2\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") " pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:46.279243 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.279199    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7816aa59-ad03-45df-9901-d7e0dcad2d8c-service-ca\") pod \"console-549b96bcc9-nl9r2\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") " pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:46.279836 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.279811    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7816aa59-ad03-45df-9901-d7e0dcad2d8c-oauth-serving-cert\") pod \"console-549b96bcc9-nl9r2\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") " pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:46.279935 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.279811    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7816aa59-ad03-45df-9901-d7e0dcad2d8c-console-config\") pod \"console-549b96bcc9-nl9r2\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") " pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:46.280066 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.280043    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7816aa59-ad03-45df-9901-d7e0dcad2d8c-service-ca\") pod \"console-549b96bcc9-nl9r2\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") " pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:46.281621 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.281600    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7816aa59-ad03-45df-9901-d7e0dcad2d8c-console-oauth-config\") pod \"console-549b96bcc9-nl9r2\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") " pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:46.281721 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.281674    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7816aa59-ad03-45df-9901-d7e0dcad2d8c-console-serving-cert\") pod \"console-549b96bcc9-nl9r2\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") " pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:46.291968 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.291941    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-722q2\" (UniqueName: \"kubernetes.io/projected/7816aa59-ad03-45df-9901-d7e0dcad2d8c-kube-api-access-722q2\") pod \"console-549b96bcc9-nl9r2\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") " pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:46.380365 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.380326    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n27qs\" (UniqueName: \"kubernetes.io/projected/7a1c2e51-24a8-49a2-ae10-75ce216f38f8-kube-api-access-n27qs\") pod \"downloads-5bc67dcd46-stqc8\" (UID: \"7a1c2e51-24a8-49a2-ae10-75ce216f38f8\") " pod="openshift-console/downloads-5bc67dcd46-stqc8"
Jul 17 18:37:46.395820 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.395787    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n27qs\" (UniqueName: \"kubernetes.io/projected/7a1c2e51-24a8-49a2-ae10-75ce216f38f8-kube-api-access-n27qs\") pod \"downloads-5bc67dcd46-stqc8\" (UID: \"7a1c2e51-24a8-49a2-ae10-75ce216f38f8\") " pod="openshift-console/downloads-5bc67dcd46-stqc8"
Jul 17 18:37:46.405447 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.405423    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:46.428579 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.428543    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-stqc8"
Jul 17 18:37:46.570248 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.570194    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-549b96bcc9-nl9r2"]
Jul 17 18:37:46.572875 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:37:46.572850    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7816aa59_ad03_45df_9901_d7e0dcad2d8c.slice/crio-8ab14c6a775f305199efd044390f5c23dd9f7512f7782e683603c618b172e922 WatchSource:0}: Error finding container 8ab14c6a775f305199efd044390f5c23dd9f7512f7782e683603c618b172e922: Status 404 returned error can't find the container with id 8ab14c6a775f305199efd044390f5c23dd9f7512f7782e683603c618b172e922
Jul 17 18:37:46.588232 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:46.588205    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-stqc8"]
Jul 17 18:37:46.592465 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:37:46.592438    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1c2e51_24a8_49a2_ae10_75ce216f38f8.slice/crio-384921ecc2502faef90f0b618ec02692396b3bc3763b982c6c6dbf4ef5a4c4b5 WatchSource:0}: Error finding container 384921ecc2502faef90f0b618ec02692396b3bc3763b982c6c6dbf4ef5a4c4b5: Status 404 returned error can't find the container with id 384921ecc2502faef90f0b618ec02692396b3bc3763b982c6c6dbf4ef5a4c4b5
Jul 17 18:37:47.010358 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:47.010033    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-stqc8" event={"ID":"7a1c2e51-24a8-49a2-ae10-75ce216f38f8","Type":"ContainerStarted","Data":"384921ecc2502faef90f0b618ec02692396b3bc3763b982c6c6dbf4ef5a4c4b5"}
Jul 17 18:37:47.012437 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:47.012408    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzs5r" event={"ID":"a85d465e-b514-4d92-9869-898394cd3652","Type":"ContainerStarted","Data":"1953667d9c223d0d9f4df6b11206fdba9453ec04b7fb9835287a2ea714007627"}
Jul 17 18:37:47.013803 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:47.013776    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549b96bcc9-nl9r2" event={"ID":"7816aa59-ad03-45df-9901-d7e0dcad2d8c","Type":"ContainerStarted","Data":"8ab14c6a775f305199efd044390f5c23dd9f7512f7782e683603c618b172e922"}
Jul 17 18:37:48.499484 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:48.499381    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-xhz5l\" (UID: \"39a38b78-ab33-411b-9a10-4cd932d6c535\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l"
Jul 17 18:37:48.503462 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:48.502628    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/39a38b78-ab33-411b-9a10-4cd932d6c535-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-xhz5l\" (UID: \"39a38b78-ab33-411b-9a10-4cd932d6c535\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l"
Jul 17 18:37:48.785592 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:48.785507    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l"
Jul 17 18:37:51.599995 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.599922    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l"]
Jul 17 18:37:51.605413 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:37:51.605378    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a38b78_ab33_411b_9a10_4cd932d6c535.slice/crio-d926d2265302e012e6f050783de68cff9b2a5b64d8187ff2b0d388a609dca11c WatchSource:0}: Error finding container d926d2265302e012e6f050783de68cff9b2a5b64d8187ff2b0d388a609dca11c: Status 404 returned error can't find the container with id d926d2265302e012e6f050783de68cff9b2a5b64d8187ff2b0d388a609dca11c
Jul 17 18:37:51.608612 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:37:51.608578    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7drcz" podUID="ab87a034-e286-4b47-8513-9fb3c0b53084"
Jul 17 18:37:51.704803 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.704723    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d8675d6f7-pljn5"]
Jul 17 18:37:51.708359 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.708337    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.719939 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.719913    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 17 18:37:51.725908 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.725871    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d8675d6f7-pljn5"]
Jul 17 18:37:51.829295 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.829255    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frxl8\" (UniqueName: \"kubernetes.io/projected/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-kube-api-access-frxl8\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.829467 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.829317    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-oauth-serving-cert\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.829467 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.829374    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-console-serving-cert\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.829557 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.829496    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-trusted-ca-bundle\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.829557 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.829526    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-service-ca\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.829673 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.829595    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-console-oauth-config\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.829673 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.829647    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-console-config\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.930748 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.930714    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-console-oauth-config\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.930935 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.930773    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-console-config\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.930935 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.930816    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frxl8\" (UniqueName: \"kubernetes.io/projected/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-kube-api-access-frxl8\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.930935 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.930842    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-oauth-serving-cert\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.930935 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.930864    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-console-serving-cert\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.930935 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.930900    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-trusted-ca-bundle\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.930935 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.930922    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-service-ca\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.931812 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.931789    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-console-config\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.931812 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.931798    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-service-ca\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.932166 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.932129    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-trusted-ca-bundle\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.932418 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.932401    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-oauth-serving-cert\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.933768 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.933742    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-console-oauth-config\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.933920 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.933902    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-console-serving-cert\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:51.945448 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:51.945419    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frxl8\" (UniqueName: \"kubernetes.io/projected/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-kube-api-access-frxl8\") pod \"console-5d8675d6f7-pljn5\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") " pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:52.019557 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:52.019474    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:37:52.038244 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:52.038208    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzs5r" event={"ID":"a85d465e-b514-4d92-9869-898394cd3652","Type":"ContainerStarted","Data":"7b97fbad59b7b4829656c6e86eb04254afa714fe30d8ceb6eda5f0c840142937"}
Jul 17 18:37:52.039722 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:52.039697    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549b96bcc9-nl9r2" event={"ID":"7816aa59-ad03-45df-9901-d7e0dcad2d8c","Type":"ContainerStarted","Data":"1fc04d5a41e886442e72b4b38956621e5745f7aaebd592daf77606d4640d3428"}
Jul 17 18:37:52.040859 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:52.040838    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7drcz"
Jul 17 18:37:52.040964 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:52.040866    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l" event={"ID":"39a38b78-ab33-411b-9a10-4cd932d6c535","Type":"ContainerStarted","Data":"d926d2265302e012e6f050783de68cff9b2a5b64d8187ff2b0d388a609dca11c"}
Jul 17 18:37:52.067335 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:52.065986    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bzs5r" podStartSLOduration=17.219792137 podStartE2EDuration="31.065965269s" totalImagesPullingTime="13.846173132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:37:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:52.065081679 +0000 UTC m=+282.506590364" watchObservedRunningTime="2026-07-17 18:37:52.065965269 +0000 UTC m=+282.507473958"
Jul 17 18:37:52.098089 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:52.098039    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-549b96bcc9-nl9r2" podStartSLOduration=1.231627623 podStartE2EDuration="6.09802035s" totalImagesPullingTime="4.866392727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:52.09759728 +0000 UTC m=+282.539105969" watchObservedRunningTime="2026-07-17 18:37:52.09802035 +0000 UTC m=+282.539529038"
Jul 17 18:37:52.165004 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:52.164929    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d8675d6f7-pljn5"]
Jul 17 18:37:52.169608 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:37:52.169585    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a84a61_e7f2_4cb4_b05e_9fb62fa2ab02.slice/crio-a88b1f0499a8dd67bdb8d5d2fefd2cf3cbd298c705359d8997c15fdad8c7514d WatchSource:0}: Error finding container a88b1f0499a8dd67bdb8d5d2fefd2cf3cbd298c705359d8997c15fdad8c7514d: Status 404 returned error can't find the container with id a88b1f0499a8dd67bdb8d5d2fefd2cf3cbd298c705359d8997c15fdad8c7514d
Jul 17 18:37:53.047473 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:53.047399    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8675d6f7-pljn5" event={"ID":"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02","Type":"ContainerStarted","Data":"3f5e845f90b77b7967401a853589a558d61633db372c1f333b961818ea2d9a5f"}
Jul 17 18:37:53.047473 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:53.047460    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8675d6f7-pljn5" event={"ID":"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02","Type":"ContainerStarted","Data":"a88b1f0499a8dd67bdb8d5d2fefd2cf3cbd298c705359d8997c15fdad8c7514d"}
Jul 17 18:37:53.066387 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:53.066326    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d8675d6f7-pljn5" podStartSLOduration=2.066290174 podStartE2EDuration="2.066290174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:53.066240525 +0000 UTC m=+283.507749258" watchObservedRunningTime="2026-07-17 18:37:53.066290174 +0000 UTC m=+283.507798864"
Jul 17 18:37:54.856751 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:54.856710    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:37:54.859597 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:54.859571    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab87a034-e286-4b47-8513-9fb3c0b53084-metrics-tls\") pod \"dns-default-7drcz\" (UID: \"ab87a034-e286-4b47-8513-9fb3c0b53084\") " pod="openshift-dns/dns-default-7drcz"
Jul 17 18:37:55.044929 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:55.044897    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vz2rf\""
Jul 17 18:37:55.052275 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:55.052243    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7drcz"
Jul 17 18:37:55.056548 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:55.056515    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l" event={"ID":"39a38b78-ab33-411b-9a10-4cd932d6c535","Type":"ContainerStarted","Data":"fff4d2253f1dd8ede2b18385a392676a1ab8915ee15846851d2e31f10cb22cea"}
Jul 17 18:37:55.094243 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:55.094192    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-xhz5l" podStartSLOduration=36.507500526 podStartE2EDuration="39.094175981s" totalImagesPullingTime="2.586675455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:55.093288181 +0000 UTC m=+285.534796871" watchObservedRunningTime="2026-07-17 18:37:55.094175981 +0000 UTC m=+285.535684669"
Jul 17 18:37:55.230289 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:55.230250    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7drcz"]
Jul 17 18:37:55.234056 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:37:55.234024    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab87a034_e286_4b47_8513_9fb3c0b53084.slice/crio-86d70744c7917928a892e9958a8556b988c660d436689f3d4ed02c6cab747fd6 WatchSource:0}: Error finding container 86d70744c7917928a892e9958a8556b988c660d436689f3d4ed02c6cab747fd6: Status 404 returned error can't find the container with id 86d70744c7917928a892e9958a8556b988c660d436689f3d4ed02c6cab747fd6
Jul 17 18:37:56.063424 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:56.063384    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7drcz" event={"ID":"ab87a034-e286-4b47-8513-9fb3c0b53084","Type":"ContainerStarted","Data":"86d70744c7917928a892e9958a8556b988c660d436689f3d4ed02c6cab747fd6"}
Jul 17 18:37:56.406490 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:56.406395    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:56.406653 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:56.406491    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:37:56.407827 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:56.407800    2559 patch_prober.go:28] interesting pod/console-549b96bcc9-nl9r2 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection refused" start-of-body=
Jul 17 18:37:56.407955 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:56.407849    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-549b96bcc9-nl9r2" podUID="7816aa59-ad03-45df-9901-d7e0dcad2d8c" containerName="console" probeResult="failure" output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection refused"
Jul 17 18:37:58.074986 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:58.074942    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7drcz" event={"ID":"ab87a034-e286-4b47-8513-9fb3c0b53084","Type":"ContainerStarted","Data":"21ce64fa506306012b345e49d30640d3a4e8242737aca6210251602f76a69754"}
Jul 17 18:37:58.075447 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:58.075045    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7drcz" event={"ID":"ab87a034-e286-4b47-8513-9fb3c0b53084","Type":"ContainerStarted","Data":"8fe4cfdc5412bb143084e4159448d39d752f78f9926d5425751bd65acf413679"}
Jul 17 18:37:58.075447 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:37:58.075078    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7drcz"
Jul 17 18:38:02.019878 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.019823    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:38:02.020399 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.019901    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:38:02.021397 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.021368    2559 patch_prober.go:28] interesting pod/console-5d8675d6f7-pljn5 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.19:8443/health\": dial tcp 10.133.0.19:8443: connect: connection refused" start-of-body=
Jul 17 18:38:02.021517 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.021412    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5d8675d6f7-pljn5" podUID="d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02" containerName="console" probeResult="failure" output="Get \"https://10.133.0.19:8443/health\": dial tcp 10.133.0.19:8443: connect: connection refused"
Jul 17 18:38:02.550507 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.550442    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7drcz" podStartSLOduration=256.810490693 podStartE2EDuration="4m18.550425056s" totalImagesPullingTime="1.739934363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:58.101229928 +0000 UTC m=+288.542738623" watchObservedRunningTime="2026-07-17 18:38:02.550425056 +0000 UTC m=+292.991933723"
Jul 17 18:38:02.551162 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.551110    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lzpxc"]
Jul 17 18:38:02.580450 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.580410    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"]
Jul 17 18:38:02.580646 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.580553    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.584490 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.583289    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 17 18:38:02.584490 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.583737    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bkwv8\""
Jul 17 18:38:02.584490 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.583924    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 17 18:38:02.584490 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.584114    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 17 18:38:02.584490 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.584351    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 17 18:38:02.589917 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.589760    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"
Jul 17 18:38:02.591056 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.590245    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"]
Jul 17 18:38:02.594627 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.593070    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 17 18:38:02.594627 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.593296    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 17 18:38:02.594627 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.593482    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wcgxm\""
Jul 17 18:38:02.594627 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.594474    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 17 18:38:02.624358 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.624300    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/84cadd37-39b2-4702-95f3-b4cb570e8ba6-node-exporter-textfile\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.624532 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.624378    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84cadd37-39b2-4702-95f3-b4cb570e8ba6-node-exporter-tls\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.624532 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.624459    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84cadd37-39b2-4702-95f3-b4cb570e8ba6-root\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.624532 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.624508    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9306cc45-d78b-47ea-846a-3c0f881b8df8-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-8pwvw\" (UID: \"9306cc45-d78b-47ea-846a-3c0f881b8df8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"
Jul 17 18:38:02.624532 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.624529    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84cadd37-39b2-4702-95f3-b4cb570e8ba6-node-exporter-wtmp\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.624727 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.624577    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9306cc45-d78b-47ea-846a-3c0f881b8df8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-8pwvw\" (UID: \"9306cc45-d78b-47ea-846a-3c0f881b8df8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"
Jul 17 18:38:02.624727 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.624606    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24bk2\" (UniqueName: \"kubernetes.io/projected/84cadd37-39b2-4702-95f3-b4cb570e8ba6-kube-api-access-24bk2\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.624727 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.624638    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9306cc45-d78b-47ea-846a-3c0f881b8df8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-8pwvw\" (UID: \"9306cc45-d78b-47ea-846a-3c0f881b8df8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"
Jul 17 18:38:02.624727 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.624685    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h67wd\" (UniqueName: \"kubernetes.io/projected/9306cc45-d78b-47ea-846a-3c0f881b8df8-kube-api-access-h67wd\") pod \"kube-state-metrics-56c5594674-8pwvw\" (UID: \"9306cc45-d78b-47ea-846a-3c0f881b8df8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"
Jul 17 18:38:02.624727 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.624715    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9306cc45-d78b-47ea-846a-3c0f881b8df8-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-8pwvw\" (UID: \"9306cc45-d78b-47ea-846a-3c0f881b8df8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"
Jul 17 18:38:02.624958 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.624748    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9306cc45-d78b-47ea-846a-3c0f881b8df8-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-8pwvw\" (UID: \"9306cc45-d78b-47ea-846a-3c0f881b8df8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"
Jul 17 18:38:02.624958 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.624787    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84cadd37-39b2-4702-95f3-b4cb570e8ba6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.624958 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.624844    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84cadd37-39b2-4702-95f3-b4cb570e8ba6-metrics-client-ca\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.624958 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.624870    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/84cadd37-39b2-4702-95f3-b4cb570e8ba6-node-exporter-accelerators-collector-config\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.624958 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.624892    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84cadd37-39b2-4702-95f3-b4cb570e8ba6-sys\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.726112 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.726077    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/84cadd37-39b2-4702-95f3-b4cb570e8ba6-node-exporter-textfile\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.726112 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.726120    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84cadd37-39b2-4702-95f3-b4cb570e8ba6-node-exporter-tls\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.726380 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.726148    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84cadd37-39b2-4702-95f3-b4cb570e8ba6-root\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.726380 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.726186    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9306cc45-d78b-47ea-846a-3c0f881b8df8-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-8pwvw\" (UID: \"9306cc45-d78b-47ea-846a-3c0f881b8df8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"
Jul 17 18:38:02.726380 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.726220    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84cadd37-39b2-4702-95f3-b4cb570e8ba6-node-exporter-wtmp\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.726380 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.726255    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84cadd37-39b2-4702-95f3-b4cb570e8ba6-root\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.726380 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:38:02.726289    2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 17 18:38:02.726611 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:38:02.726409    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84cadd37-39b2-4702-95f3-b4cb570e8ba6-node-exporter-tls podName:84cadd37-39b2-4702-95f3-b4cb570e8ba6 nodeName:}" failed. No retries permitted until 2026-07-17 18:38:03.226354304 +0000 UTC m=+293.667862977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/84cadd37-39b2-4702-95f3-b4cb570e8ba6-node-exporter-tls") pod "node-exporter-lzpxc" (UID: "84cadd37-39b2-4702-95f3-b4cb570e8ba6") : secret "node-exporter-tls" not found
Jul 17 18:38:02.726667 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.726618    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/84cadd37-39b2-4702-95f3-b4cb570e8ba6-node-exporter-textfile\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.726667 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.726256    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9306cc45-d78b-47ea-846a-3c0f881b8df8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-8pwvw\" (UID: \"9306cc45-d78b-47ea-846a-3c0f881b8df8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"
Jul 17 18:38:02.726762 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.726676    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24bk2\" (UniqueName: \"kubernetes.io/projected/84cadd37-39b2-4702-95f3-b4cb570e8ba6-kube-api-access-24bk2\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.726762 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.726705    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9306cc45-d78b-47ea-846a-3c0f881b8df8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-8pwvw\" (UID: \"9306cc45-d78b-47ea-846a-3c0f881b8df8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"
Jul 17 18:38:02.726762 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.726730    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h67wd\" (UniqueName: \"kubernetes.io/projected/9306cc45-d78b-47ea-846a-3c0f881b8df8-kube-api-access-h67wd\") pod \"kube-state-metrics-56c5594674-8pwvw\" (UID: \"9306cc45-d78b-47ea-846a-3c0f881b8df8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"
Jul 17 18:38:02.727022 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.726804    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84cadd37-39b2-4702-95f3-b4cb570e8ba6-node-exporter-wtmp\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.727022 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.726925    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9306cc45-d78b-47ea-846a-3c0f881b8df8-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-8pwvw\" (UID: \"9306cc45-d78b-47ea-846a-3c0f881b8df8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"
Jul 17 18:38:02.727022 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.726989    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9306cc45-d78b-47ea-846a-3c0f881b8df8-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-8pwvw\" (UID: \"9306cc45-d78b-47ea-846a-3c0f881b8df8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"
Jul 17 18:38:02.727022 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.726997    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9306cc45-d78b-47ea-846a-3c0f881b8df8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-8pwvw\" (UID: \"9306cc45-d78b-47ea-846a-3c0f881b8df8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"
Jul 17 18:38:02.727267 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.727024    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84cadd37-39b2-4702-95f3-b4cb570e8ba6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.727267 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.727072    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84cadd37-39b2-4702-95f3-b4cb570e8ba6-metrics-client-ca\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.727267 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.727108    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/84cadd37-39b2-4702-95f3-b4cb570e8ba6-node-exporter-accelerators-collector-config\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.727267 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.727136    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84cadd37-39b2-4702-95f3-b4cb570e8ba6-sys\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.727267 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.727226    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84cadd37-39b2-4702-95f3-b4cb570e8ba6-sys\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.727528 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.727327    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9306cc45-d78b-47ea-846a-3c0f881b8df8-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-8pwvw\" (UID: \"9306cc45-d78b-47ea-846a-3c0f881b8df8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"
Jul 17 18:38:02.727825 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.727800    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/84cadd37-39b2-4702-95f3-b4cb570e8ba6-node-exporter-accelerators-collector-config\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.728244 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.728202    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84cadd37-39b2-4702-95f3-b4cb570e8ba6-metrics-client-ca\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.728915 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.728879    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9306cc45-d78b-47ea-846a-3c0f881b8df8-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-8pwvw\" (UID: \"9306cc45-d78b-47ea-846a-3c0f881b8df8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"
Jul 17 18:38:02.730157 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.730137    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9306cc45-d78b-47ea-846a-3c0f881b8df8-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-8pwvw\" (UID: \"9306cc45-d78b-47ea-846a-3c0f881b8df8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"
Jul 17 18:38:02.731076 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.731050    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84cadd37-39b2-4702-95f3-b4cb570e8ba6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.731861 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.731837    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9306cc45-d78b-47ea-846a-3c0f881b8df8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-8pwvw\" (UID: \"9306cc45-d78b-47ea-846a-3c0f881b8df8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"
Jul 17 18:38:02.736199 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.736172    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24bk2\" (UniqueName: \"kubernetes.io/projected/84cadd37-39b2-4702-95f3-b4cb570e8ba6-kube-api-access-24bk2\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:02.736727 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.736704    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h67wd\" (UniqueName: \"kubernetes.io/projected/9306cc45-d78b-47ea-846a-3c0f881b8df8-kube-api-access-h67wd\") pod \"kube-state-metrics-56c5594674-8pwvw\" (UID: \"9306cc45-d78b-47ea-846a-3c0f881b8df8\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"
Jul 17 18:38:02.909434 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:02.909397    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"
Jul 17 18:38:03.122635 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:38:03.122583    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-tt279" podUID="f9fc2de6-adf3-45c0-952a-e317f362aa51"
Jul 17 18:38:03.231503 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.231414    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84cadd37-39b2-4702-95f3-b4cb570e8ba6-node-exporter-tls\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:03.234393 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.234364    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84cadd37-39b2-4702-95f3-b4cb570e8ba6-node-exporter-tls\") pod \"node-exporter-lzpxc\" (UID: \"84cadd37-39b2-4702-95f3-b4cb570e8ba6\") " pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:03.500272 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.500196    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lzpxc"
Jul 17 18:38:03.641624 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.641588    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 17 18:38:03.685209 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.685177    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 17 18:38:03.685410 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.685363    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.689169 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.689137    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 17 18:38:03.689405 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.689388    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 17 18:38:03.689678 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.689529    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 17 18:38:03.689678 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.689524    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 17 18:38:03.689678 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.689607    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 17 18:38:03.689850 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.689726    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 17 18:38:03.690605 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.690022    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 17 18:38:03.690605 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.690236    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 17 18:38:03.690605 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.690522    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-p88sp\""
Jul 17 18:38:03.691409 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.691389    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 17 18:38:03.737261 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.737222    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-config-out\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.737465 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.737271    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.737465 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.737301    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56lzg\" (UniqueName: \"kubernetes.io/projected/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-kube-api-access-56lzg\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.737465 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.737348    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-web-config\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.737465 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.737395    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-config-volume\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.737465 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.737456    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.737734 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.737477    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.737734 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.737523    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.737734 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.737547    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.737734 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.737571    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.737734 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.737599    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.737734 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.737629    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.737734 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.737652    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.838379 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.838295    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.838379 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.838366    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.838699 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.838420    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.838699 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.838445    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.838699 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.838472    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.838699 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.838503    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.838699 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.838532    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.838699 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.838559    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.838699 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.838603    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-config-out\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.838699 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.838624    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.838699 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.838642    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56lzg\" (UniqueName: \"kubernetes.io/projected/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-kube-api-access-56lzg\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.838699 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.838661    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-web-config\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.838699 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.838694    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-config-volume\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.840407 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:38:03.839753    2559 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 17 18:38:03.840407 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:38:03.839916    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-secret-alertmanager-main-tls podName:8c076bd6-5c9d-4dc8-93de-e54c69cf8673 nodeName:}" failed. No retries permitted until 2026-07-17 18:38:04.339814708 +0000 UTC m=+294.781323375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "8c076bd6-5c9d-4dc8-93de-e54c69cf8673") : secret "alertmanager-main-tls" not found
Jul 17 18:38:03.840407 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.839925    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.841658 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.841356    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.844158 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.843888    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.844505 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.844452    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.844972 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.844953    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.845160 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.845098    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-config-volume\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.845268 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.845208    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-web-config\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.846091 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.845527    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-config-out\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.846091 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.845676    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.846091 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.846049    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.847615 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.847559    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:03.849679 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:03.849654    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56lzg\" (UniqueName: \"kubernetes.io/projected/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-kube-api-access-56lzg\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:04.344575 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:04.344492    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:04.347614 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:04.347585    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8c076bd6-5c9d-4dc8-93de-e54c69cf8673-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8c076bd6-5c9d-4dc8-93de-e54c69cf8673\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:04.598059 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:04.597974    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 17 18:38:05.067630 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:38:05.067601    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84cadd37_39b2_4702_95f3_b4cb570e8ba6.slice/crio-14af5325a6f5600cab60b2d677650e33e588fdf4335c1cee8f8d2fc6debffbc7 WatchSource:0}: Error finding container 14af5325a6f5600cab60b2d677650e33e588fdf4335c1cee8f8d2fc6debffbc7: Status 404 returned error can't find the container with id 14af5325a6f5600cab60b2d677650e33e588fdf4335c1cee8f8d2fc6debffbc7
Jul 17 18:38:05.104411 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:05.104285    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lzpxc" event={"ID":"84cadd37-39b2-4702-95f3-b4cb570e8ba6","Type":"ContainerStarted","Data":"14af5325a6f5600cab60b2d677650e33e588fdf4335c1cee8f8d2fc6debffbc7"}
Jul 17 18:38:05.204544 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:05.204517    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-8pwvw"]
Jul 17 18:38:05.232926 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:05.232905    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 17 18:38:05.343182 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:38:05.343098    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9306cc45_d78b_47ea_846a_3c0f881b8df8.slice/crio-fea1c533a41ec877d14eb013f904fd2e801c9acc213b5f037320e5bbdc1926bd WatchSource:0}: Error finding container fea1c533a41ec877d14eb013f904fd2e801c9acc213b5f037320e5bbdc1926bd: Status 404 returned error can't find the container with id fea1c533a41ec877d14eb013f904fd2e801c9acc213b5f037320e5bbdc1926bd
Jul 17 18:38:05.343855 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:38:05.343810    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c076bd6_5c9d_4dc8_93de_e54c69cf8673.slice/crio-b26d8bcd2b8292282c3d4563e09768716967ccbc7e3e8eb3d51205fe4eb430ac WatchSource:0}: Error finding container b26d8bcd2b8292282c3d4563e09768716967ccbc7e3e8eb3d51205fe4eb430ac: Status 404 returned error can't find the container with id b26d8bcd2b8292282c3d4563e09768716967ccbc7e3e8eb3d51205fe4eb430ac
Jul 17 18:38:06.112336 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:06.111974    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-stqc8" event={"ID":"7a1c2e51-24a8-49a2-ae10-75ce216f38f8","Type":"ContainerStarted","Data":"ac861ba1dbea86a5feae797703be3b0da9ec0e57cda3df4086f148841d336f08"}
Jul 17 18:38:06.115144 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:06.115087    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-stqc8"
Jul 17 18:38:06.117566 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:06.117502    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw" event={"ID":"9306cc45-d78b-47ea-846a-3c0f881b8df8","Type":"ContainerStarted","Data":"fea1c533a41ec877d14eb013f904fd2e801c9acc213b5f037320e5bbdc1926bd"}
Jul 17 18:38:06.131857 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:06.131736    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-stqc8"
Jul 17 18:38:06.131857 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:06.131786    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c076bd6-5c9d-4dc8-93de-e54c69cf8673","Type":"ContainerStarted","Data":"b26d8bcd2b8292282c3d4563e09768716967ccbc7e3e8eb3d51205fe4eb430ac"}
Jul 17 18:38:06.140442 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:06.138343    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-stqc8" podStartSLOduration=1.32422517 podStartE2EDuration="20.138323724s" totalImagesPullingTime="18.814098554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:38:06.13780111 +0000 UTC m=+296.579309800" watchObservedRunningTime="2026-07-17 18:38:06.138323724 +0000 UTC m=+296.579832411"
Jul 17 18:38:06.406380 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:06.406225    2559 patch_prober.go:28] interesting pod/console-549b96bcc9-nl9r2 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection refused" start-of-body=
Jul 17 18:38:06.406966 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:06.406295    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-549b96bcc9-nl9r2" podUID="7816aa59-ad03-45df-9901-d7e0dcad2d8c" containerName="console" probeResult="failure" output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection refused"
Jul 17 18:38:08.080819 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:08.080794    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7drcz"
Jul 17 18:38:08.132752 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:08.132716    2559 generic.go:356] "Generic (PLEG): container finished" podID="5489e8dd-e1cb-4247-88a4-4c2d44f0633c" containerID="f3d3cabe49a07a2c92b79e141f8d4fb62390981c7f1d7b4f0c8e091903253c43" exitCode=255
Jul 17 18:38:08.132942 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:08.132795    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8" event={"ID":"5489e8dd-e1cb-4247-88a4-4c2d44f0633c","Type":"ContainerDied","Data":"f3d3cabe49a07a2c92b79e141f8d4fb62390981c7f1d7b4f0c8e091903253c43"}
Jul 17 18:38:08.132942 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:08.132859    2559 scope.go:117] "RemoveContainer" containerID="ac5641a621a90624a790a92cdc1d344a3f8a0f5dbde92978b248c3e8ea4f6007"
Jul 17 18:38:08.140372 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:08.140351    2559 scope.go:117] "RemoveContainer" containerID="f3d3cabe49a07a2c92b79e141f8d4fb62390981c7f1d7b4f0c8e091903253c43"
Jul 17 18:38:08.140539 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:38:08.140520    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-6fd976888b-6q6q8_open-cluster-management-agent-addon(5489e8dd-e1cb-4247-88a4-4c2d44f0633c)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8" podUID="5489e8dd-e1cb-4247-88a4-4c2d44f0633c"
Jul 17 18:38:08.217551 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:08.217519    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-549b96bcc9-nl9r2"]
Jul 17 18:38:09.120785 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:09.120744    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8"
Jul 17 18:38:09.139763 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:09.139725    2559 generic.go:356] "Generic (PLEG): container finished" podID="8c076bd6-5c9d-4dc8-93de-e54c69cf8673" containerID="d6a2cceea1a8ed3c646df8aeef649adcecc2cdf8bd3b9e9a503e6eeec668b7c0" exitCode=0
Jul 17 18:38:09.139955 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:09.139814    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c076bd6-5c9d-4dc8-93de-e54c69cf8673","Type":"ContainerDied","Data":"d6a2cceea1a8ed3c646df8aeef649adcecc2cdf8bd3b9e9a503e6eeec668b7c0"}
Jul 17 18:38:09.141769 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:09.141740    2559 generic.go:356] "Generic (PLEG): container finished" podID="84cadd37-39b2-4702-95f3-b4cb570e8ba6" containerID="0d5650e9302174bf4f2031e123bc543673876c9e3116e331d97c270962684ffb" exitCode=0
Jul 17 18:38:09.141845 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:09.141818    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lzpxc" event={"ID":"84cadd37-39b2-4702-95f3-b4cb570e8ba6","Type":"ContainerDied","Data":"0d5650e9302174bf4f2031e123bc543673876c9e3116e331d97c270962684ffb"}
Jul 17 18:38:09.144667 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:09.143946    2559 scope.go:117] "RemoveContainer" containerID="f3d3cabe49a07a2c92b79e141f8d4fb62390981c7f1d7b4f0c8e091903253c43"
Jul 17 18:38:09.144667 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:38:09.144096    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-6fd976888b-6q6q8_open-cluster-management-agent-addon(5489e8dd-e1cb-4247-88a4-4c2d44f0633c)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8" podUID="5489e8dd-e1cb-4247-88a4-4c2d44f0633c"
Jul 17 18:38:09.151006 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:09.147015    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw" event={"ID":"9306cc45-d78b-47ea-846a-3c0f881b8df8","Type":"ContainerStarted","Data":"dfc810c04d565ddd07a6f976df1bafb40f34199a02eaf064d2a4042561810d98"}
Jul 17 18:38:09.151006 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:09.147201    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw" event={"ID":"9306cc45-d78b-47ea-846a-3c0f881b8df8","Type":"ContainerStarted","Data":"492336f21f28bd956758198592aebe724d24365b58317d7f7ac8d2857d2635fa"}
Jul 17 18:38:09.151006 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:09.147240    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw" event={"ID":"9306cc45-d78b-47ea-846a-3c0f881b8df8","Type":"ContainerStarted","Data":"d41e45bd2fa10af51d45a20473ecedf52c16b70ff41da731250039152cad1255"}
Jul 17 18:38:09.235138 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:09.235083    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-8pwvw" podStartSLOduration=4.290777248 podStartE2EDuration="7.235067048s" totalImagesPullingTime="2.9442898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:38:09.234764037 +0000 UTC m=+299.676272727" watchObservedRunningTime="2026-07-17 18:38:09.235067048 +0000 UTC m=+299.676575758"
Jul 17 18:38:09.994793 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:09.994761    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 18:38:09.999342 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:09.999292    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 18:38:10.014447 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:10.014422    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 18:38:10.016275 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:10.016251    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 18:38:10.019728 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:10.019712    2559 kubelet.go:1648] "Image garbage collection succeeded"
Jul 17 18:38:10.154266 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:10.153587    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lzpxc" event={"ID":"84cadd37-39b2-4702-95f3-b4cb570e8ba6","Type":"ContainerStarted","Data":"a7ea98c88d83a2304182ccf40c312cd9768d3ef323f932508eebbe89eb4e4a7d"}
Jul 17 18:38:10.154266 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:10.153658    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lzpxc" event={"ID":"84cadd37-39b2-4702-95f3-b4cb570e8ba6","Type":"ContainerStarted","Data":"1faffd2325e328bd4fe980d4b772506fec2b69e328cd3fbbb7548c5d4b8f0abe"}
Jul 17 18:38:10.331037 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:10.184997    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lzpxc" podStartSLOduration=4.80360645 podStartE2EDuration="8.184979728s" totalImagesPullingTime="3.229644658s" totalInitContainerRuntime="151.72862ms" isStatefulPod=true podCreationTimestamp="2026-07-17 18:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:38:10.184805652 +0000 UTC m=+300.626314354" watchObservedRunningTime="2026-07-17 18:38:10.184979728 +0000 UTC m=+300.626488614"
Jul 17 18:38:12.024649 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:12.024619    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:38:12.029947 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:12.029925    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:38:12.169620 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:12.169584    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c076bd6-5c9d-4dc8-93de-e54c69cf8673","Type":"ContainerStarted","Data":"03800a9bb619581908af12b45ff029494bb1f47c919e99f2743db2f3dbbc5356"}
Jul 17 18:38:12.169779 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:12.169679    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c076bd6-5c9d-4dc8-93de-e54c69cf8673","Type":"ContainerStarted","Data":"78e1076f322b2232cdef3c2ff36f69aac2f79292d2381ff732349c9c3cfc06a1"}
Jul 17 18:38:12.169779 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:12.169723    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c076bd6-5c9d-4dc8-93de-e54c69cf8673","Type":"ContainerStarted","Data":"ad109745403396b0aa57d763a619948949a3beebe7ed120c2452115b5883957a"}
Jul 17 18:38:12.169779 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:12.169760    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c076bd6-5c9d-4dc8-93de-e54c69cf8673","Type":"ContainerStarted","Data":"a92646fcfb4a85ca24219bc17ea8172c28da42bf14a3c09355e84777381da33e"}
Jul 17 18:38:12.298747 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:12.298722    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 18:38:13.180001 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:13.179939    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c076bd6-5c9d-4dc8-93de-e54c69cf8673","Type":"ContainerStarted","Data":"f0b3914a85f47403e218310814bfe3f92bd26c695a8656dfe8a195197af04ebd"}
Jul 17 18:38:14.189085 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:14.189044    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c076bd6-5c9d-4dc8-93de-e54c69cf8673","Type":"ContainerStarted","Data":"c46ac7f8adf1abda72122ef0019214cb74849a53462c3dafa95a00afe3ea01b0"}
Jul 17 18:38:14.221890 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:14.221830    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.309363111 podStartE2EDuration="11.221809973s" totalImagesPullingTime="6.89328736s" totalInitContainerRuntime="19.159502ms" isStatefulPod=false podCreationTimestamp="2026-07-17 18:38:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:38:14.219527494 +0000 UTC m=+304.661036183" watchObservedRunningTime="2026-07-17 18:38:14.221809973 +0000 UTC m=+304.663318662"
Jul 17 18:38:18.119558 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:18.119524    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:38:18.760539 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:18.760492    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d8675d6f7-pljn5"]
Jul 17 18:38:21.125399 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:21.125378    2559 scope.go:117] "RemoveContainer" containerID="f3d3cabe49a07a2c92b79e141f8d4fb62390981c7f1d7b4f0c8e091903253c43"
Jul 17 18:38:22.228977 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:22.228940    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd976888b-6q6q8" event={"ID":"5489e8dd-e1cb-4247-88a4-4c2d44f0633c","Type":"ContainerStarted","Data":"45536e26318eec486457443b90954648c0b1746cdce5f7838ca5a4fca3236625"}
Jul 17 18:38:31.264364 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:31.264329    2559 generic.go:356] "Generic (PLEG): container finished" podID="b92733f1-3721-44de-a7d9-c19deb63d6a4" containerID="9e799d84400c0bfb99b3301b4cde83d2a9492fc451d7bf7d982d31dd5f0cfe70" exitCode=0
Jul 17 18:38:31.264756 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:31.264400    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-tmwrz" event={"ID":"b92733f1-3721-44de-a7d9-c19deb63d6a4","Type":"ContainerDied","Data":"9e799d84400c0bfb99b3301b4cde83d2a9492fc451d7bf7d982d31dd5f0cfe70"}
Jul 17 18:38:31.264797 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:31.264757    2559 scope.go:117] "RemoveContainer" containerID="9e799d84400c0bfb99b3301b4cde83d2a9492fc451d7bf7d982d31dd5f0cfe70"
Jul 17 18:38:31.847103 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:31.847071    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lfvbt_0881b192-5909-49a5-ba16-65dbc6829189/serve-healthcheck-canary/0.log"
Jul 17 18:38:32.269770 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:32.269689    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-tmwrz" event={"ID":"b92733f1-3721-44de-a7d9-c19deb63d6a4","Type":"ContainerStarted","Data":"6e33c3927eaaa35fbab894ffe4a0660f86e0477d3da05619b89fc3f7cd3724a7"}
Jul 17 18:38:33.242398 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:33.242342    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-549b96bcc9-nl9r2" podUID="7816aa59-ad03-45df-9901-d7e0dcad2d8c" containerName="console" containerID="cri-o://1fc04d5a41e886442e72b4b38956621e5745f7aaebd592daf77606d4640d3428" gracePeriod=15
Jul 17 18:38:33.520755 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:33.520728    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:38:33.624557 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:33.624528    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7816aa59-ad03-45df-9901-d7e0dcad2d8c-console-config\") pod \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") "
Jul 17 18:38:33.624557 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:33.624561    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7816aa59-ad03-45df-9901-d7e0dcad2d8c-service-ca\") pod \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") "
Jul 17 18:38:33.624769 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:33.624625    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-722q2\" (UniqueName: \"kubernetes.io/projected/7816aa59-ad03-45df-9901-d7e0dcad2d8c-kube-api-access-722q2\") pod \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") "
Jul 17 18:38:33.624769 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:33.624657    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7816aa59-ad03-45df-9901-d7e0dcad2d8c-console-oauth-config\") pod \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") "
Jul 17 18:38:33.624769 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:33.624688    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7816aa59-ad03-45df-9901-d7e0dcad2d8c-oauth-serving-cert\") pod \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") "
Jul 17 18:38:33.624769 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:33.624719    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7816aa59-ad03-45df-9901-d7e0dcad2d8c-console-serving-cert\") pod \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\" (UID: \"7816aa59-ad03-45df-9901-d7e0dcad2d8c\") "
Jul 17 18:38:33.625053 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:33.625023    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7816aa59-ad03-45df-9901-d7e0dcad2d8c-service-ca" (OuterVolumeSpecName: "service-ca") pod "7816aa59-ad03-45df-9901-d7e0dcad2d8c" (UID: "7816aa59-ad03-45df-9901-d7e0dcad2d8c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:38:33.625201 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:33.625019    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7816aa59-ad03-45df-9901-d7e0dcad2d8c-console-config" (OuterVolumeSpecName: "console-config") pod "7816aa59-ad03-45df-9901-d7e0dcad2d8c" (UID: "7816aa59-ad03-45df-9901-d7e0dcad2d8c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:38:33.625201 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:33.625063    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7816aa59-ad03-45df-9901-d7e0dcad2d8c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7816aa59-ad03-45df-9901-d7e0dcad2d8c" (UID: "7816aa59-ad03-45df-9901-d7e0dcad2d8c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:38:33.627129 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:33.627098    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7816aa59-ad03-45df-9901-d7e0dcad2d8c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7816aa59-ad03-45df-9901-d7e0dcad2d8c" (UID: "7816aa59-ad03-45df-9901-d7e0dcad2d8c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:38:33.627129 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:33.627117    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7816aa59-ad03-45df-9901-d7e0dcad2d8c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7816aa59-ad03-45df-9901-d7e0dcad2d8c" (UID: "7816aa59-ad03-45df-9901-d7e0dcad2d8c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:38:33.627248 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:33.627157    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7816aa59-ad03-45df-9901-d7e0dcad2d8c-kube-api-access-722q2" (OuterVolumeSpecName: "kube-api-access-722q2") pod "7816aa59-ad03-45df-9901-d7e0dcad2d8c" (UID: "7816aa59-ad03-45df-9901-d7e0dcad2d8c"). InnerVolumeSpecName "kube-api-access-722q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:38:33.725900 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:33.725863    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-722q2\" (UniqueName: \"kubernetes.io/projected/7816aa59-ad03-45df-9901-d7e0dcad2d8c-kube-api-access-722q2\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:38:33.725900 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:33.725891    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7816aa59-ad03-45df-9901-d7e0dcad2d8c-console-oauth-config\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:38:33.725900 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:33.725900    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7816aa59-ad03-45df-9901-d7e0dcad2d8c-oauth-serving-cert\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:38:33.725900 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:33.725908    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7816aa59-ad03-45df-9901-d7e0dcad2d8c-console-serving-cert\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:38:33.726160 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:33.725917    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7816aa59-ad03-45df-9901-d7e0dcad2d8c-console-config\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:38:33.726160 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:33.725925    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7816aa59-ad03-45df-9901-d7e0dcad2d8c-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:38:34.279241 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:34.279210    2559 generic.go:356] "Generic (PLEG): container finished" podID="7816aa59-ad03-45df-9901-d7e0dcad2d8c" containerID="1fc04d5a41e886442e72b4b38956621e5745f7aaebd592daf77606d4640d3428" exitCode=0
Jul 17 18:38:34.279427 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:34.279256    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549b96bcc9-nl9r2"
Jul 17 18:38:34.279427 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:34.279281    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549b96bcc9-nl9r2" event={"ID":"7816aa59-ad03-45df-9901-d7e0dcad2d8c","Type":"ContainerDied","Data":"1fc04d5a41e886442e72b4b38956621e5745f7aaebd592daf77606d4640d3428"}
Jul 17 18:38:34.279427 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:34.279355    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549b96bcc9-nl9r2" event={"ID":"7816aa59-ad03-45df-9901-d7e0dcad2d8c","Type":"ContainerDied","Data":"8ab14c6a775f305199efd044390f5c23dd9f7512f7782e683603c618b172e922"}
Jul 17 18:38:34.279427 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:34.279377    2559 scope.go:117] "RemoveContainer" containerID="1fc04d5a41e886442e72b4b38956621e5745f7aaebd592daf77606d4640d3428"
Jul 17 18:38:34.287771 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:34.287748    2559 scope.go:117] "RemoveContainer" containerID="1fc04d5a41e886442e72b4b38956621e5745f7aaebd592daf77606d4640d3428"
Jul 17 18:38:34.288066 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:38:34.288043    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fc04d5a41e886442e72b4b38956621e5745f7aaebd592daf77606d4640d3428\": container with ID starting with 1fc04d5a41e886442e72b4b38956621e5745f7aaebd592daf77606d4640d3428 not found: ID does not exist" containerID="1fc04d5a41e886442e72b4b38956621e5745f7aaebd592daf77606d4640d3428"
Jul 17 18:38:34.288142 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:34.288077    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fc04d5a41e886442e72b4b38956621e5745f7aaebd592daf77606d4640d3428"} err="failed to get container status \"1fc04d5a41e886442e72b4b38956621e5745f7aaebd592daf77606d4640d3428\": rpc error: code = NotFound desc = could not find container \"1fc04d5a41e886442e72b4b38956621e5745f7aaebd592daf77606d4640d3428\": container with ID starting with 1fc04d5a41e886442e72b4b38956621e5745f7aaebd592daf77606d4640d3428 not found: ID does not exist"
Jul 17 18:38:34.306529 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:34.306504    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-549b96bcc9-nl9r2"]
Jul 17 18:38:34.312986 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:34.312965    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-549b96bcc9-nl9r2"]
Jul 17 18:38:36.124556 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:36.124477    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7816aa59-ad03-45df-9901-d7e0dcad2d8c" path="/var/lib/kubelet/pods/7816aa59-ad03-45df-9901-d7e0dcad2d8c/volumes"
Jul 17 18:38:36.289054 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:36.289021    2559 generic.go:356] "Generic (PLEG): container finished" podID="f86765ef-cbaf-4c55-aa42-8451d8e07363" containerID="b52d3acaa16863dcc314b217f6e21780f738dcfb575ada2891419112f6dd3a5c" exitCode=0
Jul 17 18:38:36.289223 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:36.289092    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-thxzx" event={"ID":"f86765ef-cbaf-4c55-aa42-8451d8e07363","Type":"ContainerDied","Data":"b52d3acaa16863dcc314b217f6e21780f738dcfb575ada2891419112f6dd3a5c"}
Jul 17 18:38:36.289490 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:36.289474    2559 scope.go:117] "RemoveContainer" containerID="b52d3acaa16863dcc314b217f6e21780f738dcfb575ada2891419112f6dd3a5c"
Jul 17 18:38:37.294757 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:37.294725    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-thxzx" event={"ID":"f86765ef-cbaf-4c55-aa42-8451d8e07363","Type":"ContainerStarted","Data":"2af25826155946f28af147f8135e6c5fa7bb269362c325746407a84e1b9d8e3f"}
Jul 17 18:38:43.783362 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:43.783296    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d8675d6f7-pljn5" podUID="d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02" containerName="console" containerID="cri-o://3f5e845f90b77b7967401a853589a558d61633db372c1f333b961818ea2d9a5f" gracePeriod=15
Jul 17 18:38:44.085999 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.085969    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:38:44.133015 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.132970    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-service-ca" (OuterVolumeSpecName: "service-ca") pod "d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02" (UID: "d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:38:44.133219 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.132637    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-service-ca\") pod \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") "
Jul 17 18:38:44.133366 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.133271    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-console-serving-cert\") pod \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") "
Jul 17 18:38:44.133486 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.133467    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-oauth-serving-cert\") pod \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") "
Jul 17 18:38:44.133550 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.133501    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-trusted-ca-bundle\") pod \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") "
Jul 17 18:38:44.133550 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.133525    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-console-config\") pod \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") "
Jul 17 18:38:44.133723 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.133562    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frxl8\" (UniqueName: \"kubernetes.io/projected/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-kube-api-access-frxl8\") pod \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") "
Jul 17 18:38:44.133723 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.133625    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-console-oauth-config\") pod \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\" (UID: \"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02\") "
Jul 17 18:38:44.133996 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.133963    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02" (UID: "d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:38:44.134154 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.133971    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-console-config" (OuterVolumeSpecName: "console-config") pod "d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02" (UID: "d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:38:44.134449 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.134418    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02" (UID: "d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:38:44.134878 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.134855    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-oauth-serving-cert\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:38:44.134972 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.134884    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-trusted-ca-bundle\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:38:44.134972 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.134900    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-console-config\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:38:44.134972 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.134914    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:38:44.135855 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.135831    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02" (UID: "d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:38:44.136296 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.136272    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-kube-api-access-frxl8" (OuterVolumeSpecName: "kube-api-access-frxl8") pod "d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02" (UID: "d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02"). InnerVolumeSpecName "kube-api-access-frxl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:38:44.136562 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.136536    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02" (UID: "d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:38:44.236346 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.236298    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-console-serving-cert\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:38:44.236346 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.236346    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frxl8\" (UniqueName: \"kubernetes.io/projected/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-kube-api-access-frxl8\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:38:44.236541 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.236360    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02-console-oauth-config\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:38:44.331536 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.328283    2559 generic.go:356] "Generic (PLEG): container finished" podID="d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02" containerID="3f5e845f90b77b7967401a853589a558d61633db372c1f333b961818ea2d9a5f" exitCode=0
Jul 17 18:38:44.331536 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.328353    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8675d6f7-pljn5" event={"ID":"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02","Type":"ContainerDied","Data":"3f5e845f90b77b7967401a853589a558d61633db372c1f333b961818ea2d9a5f"}
Jul 17 18:38:44.331536 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.328405    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8675d6f7-pljn5" event={"ID":"d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02","Type":"ContainerDied","Data":"a88b1f0499a8dd67bdb8d5d2fefd2cf3cbd298c705359d8997c15fdad8c7514d"}
Jul 17 18:38:44.331536 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.328436    2559 scope.go:117] "RemoveContainer" containerID="3f5e845f90b77b7967401a853589a558d61633db372c1f333b961818ea2d9a5f"
Jul 17 18:38:44.331536 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.328610    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8675d6f7-pljn5"
Jul 17 18:38:44.343561 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.343525    2559 scope.go:117] "RemoveContainer" containerID="3f5e845f90b77b7967401a853589a558d61633db372c1f333b961818ea2d9a5f"
Jul 17 18:38:44.344282 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:38:44.344260    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f5e845f90b77b7967401a853589a558d61633db372c1f333b961818ea2d9a5f\": container with ID starting with 3f5e845f90b77b7967401a853589a558d61633db372c1f333b961818ea2d9a5f not found: ID does not exist" containerID="3f5e845f90b77b7967401a853589a558d61633db372c1f333b961818ea2d9a5f"
Jul 17 18:38:44.344436 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.344294    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f5e845f90b77b7967401a853589a558d61633db372c1f333b961818ea2d9a5f"} err="failed to get container status \"3f5e845f90b77b7967401a853589a558d61633db372c1f333b961818ea2d9a5f\": rpc error: code = NotFound desc = could not find container \"3f5e845f90b77b7967401a853589a558d61633db372c1f333b961818ea2d9a5f\": container with ID starting with 3f5e845f90b77b7967401a853589a558d61633db372c1f333b961818ea2d9a5f not found: ID does not exist"
Jul 17 18:38:44.356901 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.356873    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d8675d6f7-pljn5"]
Jul 17 18:38:44.360514 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:44.360488    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d8675d6f7-pljn5"]
Jul 17 18:38:46.124560 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:38:46.124532    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02" path="/var/lib/kubelet/pods/d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02/volumes"
Jul 17 18:39:23.863201 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:23.863167    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs\") pod \"network-metrics-daemon-tt279\" (UID: \"f9fc2de6-adf3-45c0-952a-e317f362aa51\") " pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:39:23.865555 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:23.865535    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fc2de6-adf3-45c0-952a-e317f362aa51-metrics-certs\") pod \"network-metrics-daemon-tt279\" (UID: \"f9fc2de6-adf3-45c0-952a-e317f362aa51\") " pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:39:24.123512 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:24.123426    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gbnj9\""
Jul 17 18:39:24.131664 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:24.131635    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tt279"
Jul 17 18:39:24.269949 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:24.269929    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tt279"]
Jul 17 18:39:24.272002 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:39:24.271978    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9fc2de6_adf3_45c0_952a_e317f362aa51.slice/crio-16bec095fd88df6b9eb6415f666a8ad31d3ef71e96cc6d2935c7e97c3ad4a40e WatchSource:0}: Error finding container 16bec095fd88df6b9eb6415f666a8ad31d3ef71e96cc6d2935c7e97c3ad4a40e: Status 404 returned error can't find the container with id 16bec095fd88df6b9eb6415f666a8ad31d3ef71e96cc6d2935c7e97c3ad4a40e
Jul 17 18:39:24.479257 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:24.479178    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tt279" event={"ID":"f9fc2de6-adf3-45c0-952a-e317f362aa51","Type":"ContainerStarted","Data":"16bec095fd88df6b9eb6415f666a8ad31d3ef71e96cc6d2935c7e97c3ad4a40e"}
Jul 17 18:39:25.484670 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:25.484575    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tt279" event={"ID":"f9fc2de6-adf3-45c0-952a-e317f362aa51","Type":"ContainerStarted","Data":"1c56ea1b1d3524afa84bda405907c3499962e90c720d0fa685c9392c10d9fbdd"}
Jul 17 18:39:25.485030 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:25.484684    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tt279" event={"ID":"f9fc2de6-adf3-45c0-952a-e317f362aa51","Type":"ContainerStarted","Data":"abb8b07439b181b1fd2fb50d3b61179275ca73853dec565515273fdbfba881b8"}
Jul 17 18:39:25.506628 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:25.506582    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tt279" podStartSLOduration=374.617886836 podStartE2EDuration="6m15.506567666s" totalImagesPullingTime="888.68083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:39:25.506090285 +0000 UTC m=+375.947598978" watchObservedRunningTime="2026-07-17 18:39:25.506567666 +0000 UTC m=+375.948076355"
Jul 17 18:39:26.965150 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.965071    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-75686874dd-fcc8n"]
Jul 17 18:39:26.965557 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.965375    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02" containerName="console"
Jul 17 18:39:26.965557 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.965386    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02" containerName="console"
Jul 17 18:39:26.965557 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.965418    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7816aa59-ad03-45df-9901-d7e0dcad2d8c" containerName="console"
Jul 17 18:39:26.965557 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.965423    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7816aa59-ad03-45df-9901-d7e0dcad2d8c" containerName="console"
Jul 17 18:39:26.965557 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.965474    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a84a61-e7f2-4cb4-b05e-9fb62fa2ab02" containerName="console"
Jul 17 18:39:26.965557 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.965484    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7816aa59-ad03-45df-9901-d7e0dcad2d8c" containerName="console"
Jul 17 18:39:26.967939 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.967919    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:26.972341 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.972301    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-b96dt\""
Jul 17 18:39:26.972447 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.972344    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 17 18:39:26.972500 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.972488    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 17 18:39:26.972683 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.972671    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 17 18:39:26.972778 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.972765    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 17 18:39:26.974713 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.974677    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 17 18:39:26.977637 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.977617    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 17 18:39:26.985240 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.985216    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:26.985355 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.985256    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:26.985355 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.985339    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-metrics-client-ca\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:26.985429 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.985359    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-telemeter-client-tls\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:26.985429 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.985378    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-secret-telemeter-client\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:26.985429 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.985396    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-federate-client-tls\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:26.985522 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.985455    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-serving-certs-ca-bundle\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:26.985522 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.985498    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt47g\" (UniqueName: \"kubernetes.io/projected/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-kube-api-access-lt47g\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:26.991118 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:26.991095    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75686874dd-fcc8n"]
Jul 17 18:39:27.086654 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:27.086618    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:27.086654 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:27.086656    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:27.086918 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:27.086718    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-metrics-client-ca\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:27.086918 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:27.086738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-telemeter-client-tls\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:27.086918 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:27.086759    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-secret-telemeter-client\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:27.086918 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:27.086782    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-federate-client-tls\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:27.086918 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:27.086807    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-serving-certs-ca-bundle\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:27.086918 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:27.086857    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt47g\" (UniqueName: \"kubernetes.io/projected/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-kube-api-access-lt47g\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:27.087753 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:27.087727    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:27.087938 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:27.087815    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-serving-certs-ca-bundle\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:27.089706 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:27.089680    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-metrics-client-ca\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:27.089867 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:27.089845    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:27.089947 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:27.089766    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-telemeter-client-tls\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:27.090229 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:27.090210    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-federate-client-tls\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:27.090324 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:27.090285    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-secret-telemeter-client\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:27.097013 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:27.096975    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt47g\" (UniqueName: \"kubernetes.io/projected/bc08b863-6a47-4d6b-b163-fb3f4fb0bda4-kube-api-access-lt47g\") pod \"telemeter-client-75686874dd-fcc8n\" (UID: \"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4\") " pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:27.277971 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:27.277886    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n"
Jul 17 18:39:27.423160 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:27.423119    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-75686874dd-fcc8n"]
Jul 17 18:39:27.424966 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:39:27.424943    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc08b863_6a47_4d6b_b163_fb3f4fb0bda4.slice/crio-9baf9a2c9abc0d1889ef1c3e363c47d1b82414ec328fa0ee9bae6d2e8036f5bc WatchSource:0}: Error finding container 9baf9a2c9abc0d1889ef1c3e363c47d1b82414ec328fa0ee9bae6d2e8036f5bc: Status 404 returned error can't find the container with id 9baf9a2c9abc0d1889ef1c3e363c47d1b82414ec328fa0ee9bae6d2e8036f5bc
Jul 17 18:39:27.493829 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:27.493795    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n" event={"ID":"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4","Type":"ContainerStarted","Data":"9baf9a2c9abc0d1889ef1c3e363c47d1b82414ec328fa0ee9bae6d2e8036f5bc"}
Jul 17 18:39:29.504543 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:29.504514    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n" event={"ID":"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4","Type":"ContainerStarted","Data":"3b84131fb98b240ac001767792c83810d3198bc852920144036f3f474edde63c"}
Jul 17 18:39:30.510672 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:30.510631    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n" event={"ID":"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4","Type":"ContainerStarted","Data":"a4b7fada30da7fb23a96af6039bc012908f471d31b9e4dcfbbbd4b5c78b42eca"}
Jul 17 18:39:30.511063 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:30.510690    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n" event={"ID":"bc08b863-6a47-4d6b-b163-fb3f4fb0bda4","Type":"ContainerStarted","Data":"3a1e939d3150bcd993aed42b93acbb4ce923b69cb3fbc8642655e464147d8726"}
Jul 17 18:39:30.536270 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:30.536214    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-75686874dd-fcc8n" podStartSLOduration=2.556644345 podStartE2EDuration="4.536198806s" totalImagesPullingTime="1.979554461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:39:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:39:30.535139334 +0000 UTC m=+380.976648039" watchObservedRunningTime="2026-07-17 18:39:30.536198806 +0000 UTC m=+380.977707495"
Jul 17 18:39:31.415738 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.415708    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c797c9845-7rj42"]
Jul 17 18:39:31.418243 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.418214    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.420844 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.420826    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 17 18:39:31.420963 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.420867    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 17 18:39:31.424183 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.424156    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nt4s4\""
Jul 17 18:39:31.424295 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.424170    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 17 18:39:31.424476 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.424459    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 17 18:39:31.424901 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.424877    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 17 18:39:31.427401 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.427386    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 17 18:39:31.434903 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.434882    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c797c9845-7rj42"]
Jul 17 18:39:31.523924 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.523891    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-console-oauth-config\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.524287 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.523939    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-console-config\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.524287 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.523984    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-trusted-ca-bundle\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.524287 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.524020    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-console-serving-cert\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.524287 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.524058    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cffqj\" (UniqueName: \"kubernetes.io/projected/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-kube-api-access-cffqj\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.524287 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.524094    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-oauth-serving-cert\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.524287 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.524143    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-service-ca\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.624734 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.624688    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-console-serving-cert\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.624734 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.624740    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cffqj\" (UniqueName: \"kubernetes.io/projected/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-kube-api-access-cffqj\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.624954 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.624775    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-oauth-serving-cert\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.625013 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.624952    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-service-ca\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.625013 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.624994    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-console-oauth-config\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.625133 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.625104    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-console-config\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.625199 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.625146    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-trusted-ca-bundle\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.626040 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.626015    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-console-config\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.626181 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.626160    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-oauth-serving-cert\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.626394 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.626374    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-service-ca\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.626494 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.626475    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-trusted-ca-bundle\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.627650 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.627625    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-console-serving-cert\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.627730 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.627649    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-console-oauth-config\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.636532 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.636510    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cffqj\" (UniqueName: \"kubernetes.io/projected/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-kube-api-access-cffqj\") pod \"console-5c797c9845-7rj42\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") " pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.727749 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.727665    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:39:31.855253 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:31.855233    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c797c9845-7rj42"]
Jul 17 18:39:31.857713 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:39:31.857689    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a39a59_0a4f_4667_a878_8a2c20f32ad6.slice/crio-6e55f4ae6396483a6dda77dbbd8f15c9edaeadf7624763049f80960ff30c655e WatchSource:0}: Error finding container 6e55f4ae6396483a6dda77dbbd8f15c9edaeadf7624763049f80960ff30c655e: Status 404 returned error can't find the container with id 6e55f4ae6396483a6dda77dbbd8f15c9edaeadf7624763049f80960ff30c655e
Jul 17 18:39:32.520069 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:32.520030    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c797c9845-7rj42" event={"ID":"b6a39a59-0a4f-4667-a878-8a2c20f32ad6","Type":"ContainerStarted","Data":"d267bd00c52ba6f98cb80299a713993b852d40674f365839e2fbae4fb6b42f5f"}
Jul 17 18:39:32.520246 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:32.520087    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c797c9845-7rj42" event={"ID":"b6a39a59-0a4f-4667-a878-8a2c20f32ad6","Type":"ContainerStarted","Data":"6e55f4ae6396483a6dda77dbbd8f15c9edaeadf7624763049f80960ff30c655e"}
Jul 17 18:39:32.545426 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:32.545368    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c797c9845-7rj42" podStartSLOduration=1.545351089 podStartE2EDuration="1.545351089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:39:32.544933342 +0000 UTC m=+382.986442031" watchObservedRunningTime="2026-07-17 18:39:32.545351089 +0000 UTC m=+382.986859782"
Jul 17 18:39:39.374243 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:39.374209    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c797c9845-7rj42"]
Jul 17 18:39:41.728001 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:39:41.727955    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:40:04.394714 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.394671    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c797c9845-7rj42" podUID="b6a39a59-0a4f-4667-a878-8a2c20f32ad6" containerName="console" containerID="cri-o://d267bd00c52ba6f98cb80299a713993b852d40674f365839e2fbae4fb6b42f5f" gracePeriod=15
Jul 17 18:40:04.642947 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.642922    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:40:04.649628 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.649570    2559 generic.go:356] "Generic (PLEG): container finished" podID="b6a39a59-0a4f-4667-a878-8a2c20f32ad6" containerID="d267bd00c52ba6f98cb80299a713993b852d40674f365839e2fbae4fb6b42f5f" exitCode=0
Jul 17 18:40:04.649628 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.649619    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c797c9845-7rj42"
Jul 17 18:40:04.649770 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.649633    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c797c9845-7rj42" event={"ID":"b6a39a59-0a4f-4667-a878-8a2c20f32ad6","Type":"ContainerDied","Data":"d267bd00c52ba6f98cb80299a713993b852d40674f365839e2fbae4fb6b42f5f"}
Jul 17 18:40:04.649770 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.649669    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c797c9845-7rj42" event={"ID":"b6a39a59-0a4f-4667-a878-8a2c20f32ad6","Type":"ContainerDied","Data":"6e55f4ae6396483a6dda77dbbd8f15c9edaeadf7624763049f80960ff30c655e"}
Jul 17 18:40:04.649770 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.649690    2559 scope.go:117] "RemoveContainer" containerID="d267bd00c52ba6f98cb80299a713993b852d40674f365839e2fbae4fb6b42f5f"
Jul 17 18:40:04.657440 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.657423    2559 scope.go:117] "RemoveContainer" containerID="d267bd00c52ba6f98cb80299a713993b852d40674f365839e2fbae4fb6b42f5f"
Jul 17 18:40:04.657719 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:40:04.657702    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d267bd00c52ba6f98cb80299a713993b852d40674f365839e2fbae4fb6b42f5f\": container with ID starting with d267bd00c52ba6f98cb80299a713993b852d40674f365839e2fbae4fb6b42f5f not found: ID does not exist" containerID="d267bd00c52ba6f98cb80299a713993b852d40674f365839e2fbae4fb6b42f5f"
Jul 17 18:40:04.657762 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.657728    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d267bd00c52ba6f98cb80299a713993b852d40674f365839e2fbae4fb6b42f5f"} err="failed to get container status \"d267bd00c52ba6f98cb80299a713993b852d40674f365839e2fbae4fb6b42f5f\": rpc error: code = NotFound desc = could not find container \"d267bd00c52ba6f98cb80299a713993b852d40674f365839e2fbae4fb6b42f5f\": container with ID starting with d267bd00c52ba6f98cb80299a713993b852d40674f365839e2fbae4fb6b42f5f not found: ID does not exist"
Jul 17 18:40:04.794665 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.794627    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-service-ca\") pod \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") "
Jul 17 18:40:04.794867 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.794675    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-trusted-ca-bundle\") pod \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") "
Jul 17 18:40:04.794867 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.794726    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-console-config\") pod \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") "
Jul 17 18:40:04.794867 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.794753    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-console-oauth-config\") pod \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") "
Jul 17 18:40:04.794867 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.794778    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-oauth-serving-cert\") pod \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") "
Jul 17 18:40:04.794867 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.794809    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cffqj\" (UniqueName: \"kubernetes.io/projected/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-kube-api-access-cffqj\") pod \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") "
Jul 17 18:40:04.794867 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.794837    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-console-serving-cert\") pod \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\" (UID: \"b6a39a59-0a4f-4667-a878-8a2c20f32ad6\") "
Jul 17 18:40:04.795210 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.795187    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b6a39a59-0a4f-4667-a878-8a2c20f32ad6" (UID: "b6a39a59-0a4f-4667-a878-8a2c20f32ad6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:40:04.795295 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.795221    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-console-config" (OuterVolumeSpecName: "console-config") pod "b6a39a59-0a4f-4667-a878-8a2c20f32ad6" (UID: "b6a39a59-0a4f-4667-a878-8a2c20f32ad6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:40:04.795499 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.795475    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-service-ca" (OuterVolumeSpecName: "service-ca") pod "b6a39a59-0a4f-4667-a878-8a2c20f32ad6" (UID: "b6a39a59-0a4f-4667-a878-8a2c20f32ad6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:40:04.795615 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.795515    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b6a39a59-0a4f-4667-a878-8a2c20f32ad6" (UID: "b6a39a59-0a4f-4667-a878-8a2c20f32ad6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:40:04.797162 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.797140    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b6a39a59-0a4f-4667-a878-8a2c20f32ad6" (UID: "b6a39a59-0a4f-4667-a878-8a2c20f32ad6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:40:04.797255 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.797206    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-kube-api-access-cffqj" (OuterVolumeSpecName: "kube-api-access-cffqj") pod "b6a39a59-0a4f-4667-a878-8a2c20f32ad6" (UID: "b6a39a59-0a4f-4667-a878-8a2c20f32ad6"). InnerVolumeSpecName "kube-api-access-cffqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:40:04.797255 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.797212    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b6a39a59-0a4f-4667-a878-8a2c20f32ad6" (UID: "b6a39a59-0a4f-4667-a878-8a2c20f32ad6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:40:04.896080 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.896047    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:40:04.896080 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.896075    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-trusted-ca-bundle\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:40:04.896080 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.896085    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-console-config\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:40:04.896328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.896093    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-console-oauth-config\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:40:04.896328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.896102    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-oauth-serving-cert\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:40:04.896328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.896110    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cffqj\" (UniqueName: \"kubernetes.io/projected/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-kube-api-access-cffqj\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:40:04.896328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.896119    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6a39a59-0a4f-4667-a878-8a2c20f32ad6-console-serving-cert\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:40:04.975601 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.975570    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c797c9845-7rj42"]
Jul 17 18:40:04.979879 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:04.979853    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c797c9845-7rj42"]
Jul 17 18:40:06.126864 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:06.126835    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6a39a59-0a4f-4667-a878-8a2c20f32ad6" path="/var/lib/kubelet/pods/b6a39a59-0a4f-4667-a878-8a2c20f32ad6/volumes"
Jul 17 18:40:42.738741 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:42.738666    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8"]
Jul 17 18:40:42.739154 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:42.738967    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a39a59-0a4f-4667-a878-8a2c20f32ad6" containerName="console"
Jul 17 18:40:42.739154 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:42.738977    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a39a59-0a4f-4667-a878-8a2c20f32ad6" containerName="console"
Jul 17 18:40:42.739154 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:42.739045    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6a39a59-0a4f-4667-a878-8a2c20f32ad6" containerName="console"
Jul 17 18:40:42.741191 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:42.741170    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8"
Jul 17 18:40:42.743900 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:42.743882    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 17 18:40:42.744052 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:42.743882    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r2ztx\""
Jul 17 18:40:42.744052 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:42.743910    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 17 18:40:42.754189 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:42.754160    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8"]
Jul 17 18:40:42.774500 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:42.774471    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30702b17-0ffa-4f3c-8904-e936284969d3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8\" (UID: \"30702b17-0ffa-4f3c-8904-e936284969d3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8"
Jul 17 18:40:42.774619 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:42.774599    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30702b17-0ffa-4f3c-8904-e936284969d3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8\" (UID: \"30702b17-0ffa-4f3c-8904-e936284969d3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8"
Jul 17 18:40:42.774660 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:42.774637    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gtm8\" (UniqueName: \"kubernetes.io/projected/30702b17-0ffa-4f3c-8904-e936284969d3-kube-api-access-9gtm8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8\" (UID: \"30702b17-0ffa-4f3c-8904-e936284969d3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8"
Jul 17 18:40:42.875958 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:42.875929    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30702b17-0ffa-4f3c-8904-e936284969d3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8\" (UID: \"30702b17-0ffa-4f3c-8904-e936284969d3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8"
Jul 17 18:40:42.876086 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:42.875965    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gtm8\" (UniqueName: \"kubernetes.io/projected/30702b17-0ffa-4f3c-8904-e936284969d3-kube-api-access-9gtm8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8\" (UID: \"30702b17-0ffa-4f3c-8904-e936284969d3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8"
Jul 17 18:40:42.876086 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:42.875994    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30702b17-0ffa-4f3c-8904-e936284969d3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8\" (UID: \"30702b17-0ffa-4f3c-8904-e936284969d3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8"
Jul 17 18:40:42.876328 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:42.876287    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30702b17-0ffa-4f3c-8904-e936284969d3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8\" (UID: \"30702b17-0ffa-4f3c-8904-e936284969d3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8"
Jul 17 18:40:42.876376 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:42.876330    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30702b17-0ffa-4f3c-8904-e936284969d3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8\" (UID: \"30702b17-0ffa-4f3c-8904-e936284969d3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8"
Jul 17 18:40:42.885222 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:42.885196    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gtm8\" (UniqueName: \"kubernetes.io/projected/30702b17-0ffa-4f3c-8904-e936284969d3-kube-api-access-9gtm8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8\" (UID: \"30702b17-0ffa-4f3c-8904-e936284969d3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8"
Jul 17 18:40:43.050151 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:43.050072    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8"
Jul 17 18:40:43.179552 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:43.179490    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8"]
Jul 17 18:40:43.181761 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:40:43.181737    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30702b17_0ffa_4f3c_8904_e936284969d3.slice/crio-9b4f11e786d095a5a4458afe92b8c3483337a1d5e42cdff1caed655be1b65edc WatchSource:0}: Error finding container 9b4f11e786d095a5a4458afe92b8c3483337a1d5e42cdff1caed655be1b65edc: Status 404 returned error can't find the container with id 9b4f11e786d095a5a4458afe92b8c3483337a1d5e42cdff1caed655be1b65edc
Jul 17 18:40:43.796020 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:43.795985    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8" event={"ID":"30702b17-0ffa-4f3c-8904-e936284969d3","Type":"ContainerStarted","Data":"9b4f11e786d095a5a4458afe92b8c3483337a1d5e42cdff1caed655be1b65edc"}
Jul 17 18:40:52.838824 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:52.838791    2559 generic.go:356] "Generic (PLEG): container finished" podID="30702b17-0ffa-4f3c-8904-e936284969d3" containerID="d5d6350d5e841b43ffc00c33f4dbe14262da866bd4778451d9d186d0e512990b" exitCode=0
Jul 17 18:40:52.839224 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:52.838841    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8" event={"ID":"30702b17-0ffa-4f3c-8904-e936284969d3","Type":"ContainerDied","Data":"d5d6350d5e841b43ffc00c33f4dbe14262da866bd4778451d9d186d0e512990b"}
Jul 17 18:40:54.850768 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:54.850735    2559 generic.go:356] "Generic (PLEG): container finished" podID="30702b17-0ffa-4f3c-8904-e936284969d3" containerID="30929407e5ff2e0b5166a170114268bcf5a3abfe5794a53abf275dcefbcec0c8" exitCode=0
Jul 17 18:40:54.850768 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:40:54.850771    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8" event={"ID":"30702b17-0ffa-4f3c-8904-e936284969d3","Type":"ContainerDied","Data":"30929407e5ff2e0b5166a170114268bcf5a3abfe5794a53abf275dcefbcec0c8"}
Jul 17 18:41:03.895375 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:03.895343    2559 generic.go:356] "Generic (PLEG): container finished" podID="30702b17-0ffa-4f3c-8904-e936284969d3" containerID="6b5c3f983b9355440b1f6e96eb9f49bc76055e9424480dbd060a933f96a1377e" exitCode=0
Jul 17 18:41:03.895784 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:03.895422    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8" event={"ID":"30702b17-0ffa-4f3c-8904-e936284969d3","Type":"ContainerDied","Data":"6b5c3f983b9355440b1f6e96eb9f49bc76055e9424480dbd060a933f96a1377e"}
Jul 17 18:41:05.075182 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:05.075157    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8"
Jul 17 18:41:05.175802 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:05.175723    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30702b17-0ffa-4f3c-8904-e936284969d3-util\") pod \"30702b17-0ffa-4f3c-8904-e936284969d3\" (UID: \"30702b17-0ffa-4f3c-8904-e936284969d3\") "
Jul 17 18:41:05.175802 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:05.175761    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gtm8\" (UniqueName: \"kubernetes.io/projected/30702b17-0ffa-4f3c-8904-e936284969d3-kube-api-access-9gtm8\") pod \"30702b17-0ffa-4f3c-8904-e936284969d3\" (UID: \"30702b17-0ffa-4f3c-8904-e936284969d3\") "
Jul 17 18:41:05.175996 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:05.175829    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30702b17-0ffa-4f3c-8904-e936284969d3-bundle\") pod \"30702b17-0ffa-4f3c-8904-e936284969d3\" (UID: \"30702b17-0ffa-4f3c-8904-e936284969d3\") "
Jul 17 18:41:05.176505 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:05.176479    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30702b17-0ffa-4f3c-8904-e936284969d3-bundle" (OuterVolumeSpecName: "bundle") pod "30702b17-0ffa-4f3c-8904-e936284969d3" (UID: "30702b17-0ffa-4f3c-8904-e936284969d3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 18:41:05.178090 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:05.178065    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30702b17-0ffa-4f3c-8904-e936284969d3-kube-api-access-9gtm8" (OuterVolumeSpecName: "kube-api-access-9gtm8") pod "30702b17-0ffa-4f3c-8904-e936284969d3" (UID: "30702b17-0ffa-4f3c-8904-e936284969d3"). InnerVolumeSpecName "kube-api-access-9gtm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:41:05.179960 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:05.179939    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30702b17-0ffa-4f3c-8904-e936284969d3-util" (OuterVolumeSpecName: "util") pod "30702b17-0ffa-4f3c-8904-e936284969d3" (UID: "30702b17-0ffa-4f3c-8904-e936284969d3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 18:41:05.277205 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:05.277173    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30702b17-0ffa-4f3c-8904-e936284969d3-util\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:41:05.277205 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:05.277201    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gtm8\" (UniqueName: \"kubernetes.io/projected/30702b17-0ffa-4f3c-8904-e936284969d3-kube-api-access-9gtm8\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:41:05.277205 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:05.277212    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30702b17-0ffa-4f3c-8904-e936284969d3-bundle\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:41:05.905063 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:05.905036    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8"
Jul 17 18:41:05.905063 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:05.905045    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzvm8" event={"ID":"30702b17-0ffa-4f3c-8904-e936284969d3","Type":"ContainerDied","Data":"9b4f11e786d095a5a4458afe92b8c3483337a1d5e42cdff1caed655be1b65edc"}
Jul 17 18:41:05.905281 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:05.905083    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b4f11e786d095a5a4458afe92b8c3483337a1d5e42cdff1caed655be1b65edc"
Jul 17 18:41:11.478580 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.478541    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7s6sp"]
Jul 17 18:41:11.479029 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.478867    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30702b17-0ffa-4f3c-8904-e936284969d3" containerName="util"
Jul 17 18:41:11.479029 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.478880    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="30702b17-0ffa-4f3c-8904-e936284969d3" containerName="util"
Jul 17 18:41:11.479029 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.478895    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30702b17-0ffa-4f3c-8904-e936284969d3" containerName="extract"
Jul 17 18:41:11.479029 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.478901    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="30702b17-0ffa-4f3c-8904-e936284969d3" containerName="extract"
Jul 17 18:41:11.479029 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.478913    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30702b17-0ffa-4f3c-8904-e936284969d3" containerName="pull"
Jul 17 18:41:11.479029 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.478918    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="30702b17-0ffa-4f3c-8904-e936284969d3" containerName="pull"
Jul 17 18:41:11.479029 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.478969    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="30702b17-0ffa-4f3c-8904-e936284969d3" containerName="extract"
Jul 17 18:41:11.480891 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.480876    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7s6sp"
Jul 17 18:41:11.486546 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.486524    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-fgbw6\""
Jul 17 18:41:11.486831 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.486817    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 17 18:41:11.487464 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.487061    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 17 18:41:11.499027 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.499003    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7s6sp"]
Jul 17 18:41:11.626229 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.626194    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/458e7dc8-c45e-42ed-abc7-b474cbbbcec5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7s6sp\" (UID: \"458e7dc8-c45e-42ed-abc7-b474cbbbcec5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7s6sp"
Jul 17 18:41:11.626721 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.626683    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pch75\" (UniqueName: \"kubernetes.io/projected/458e7dc8-c45e-42ed-abc7-b474cbbbcec5-kube-api-access-pch75\") pod \"cert-manager-operator-controller-manager-55c444bf86-7s6sp\" (UID: \"458e7dc8-c45e-42ed-abc7-b474cbbbcec5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7s6sp"
Jul 17 18:41:11.728114 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.728071    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/458e7dc8-c45e-42ed-abc7-b474cbbbcec5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7s6sp\" (UID: \"458e7dc8-c45e-42ed-abc7-b474cbbbcec5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7s6sp"
Jul 17 18:41:11.728778 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.728519    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/458e7dc8-c45e-42ed-abc7-b474cbbbcec5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7s6sp\" (UID: \"458e7dc8-c45e-42ed-abc7-b474cbbbcec5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7s6sp"
Jul 17 18:41:11.728972 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.728952    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pch75\" (UniqueName: \"kubernetes.io/projected/458e7dc8-c45e-42ed-abc7-b474cbbbcec5-kube-api-access-pch75\") pod \"cert-manager-operator-controller-manager-55c444bf86-7s6sp\" (UID: \"458e7dc8-c45e-42ed-abc7-b474cbbbcec5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7s6sp"
Jul 17 18:41:11.749787 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.749764    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pch75\" (UniqueName: \"kubernetes.io/projected/458e7dc8-c45e-42ed-abc7-b474cbbbcec5-kube-api-access-pch75\") pod \"cert-manager-operator-controller-manager-55c444bf86-7s6sp\" (UID: \"458e7dc8-c45e-42ed-abc7-b474cbbbcec5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7s6sp"
Jul 17 18:41:11.789745 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.789707    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7s6sp"
Jul 17 18:41:11.927894 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:11.927869    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7s6sp"]
Jul 17 18:41:11.931671 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:41:11.931646    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458e7dc8_c45e_42ed_abc7_b474cbbbcec5.slice/crio-7b46f2d9425e50b0ba8ee8a67672c75f002943f938aa4d8559b49258c3da2654 WatchSource:0}: Error finding container 7b46f2d9425e50b0ba8ee8a67672c75f002943f938aa4d8559b49258c3da2654: Status 404 returned error can't find the container with id 7b46f2d9425e50b0ba8ee8a67672c75f002943f938aa4d8559b49258c3da2654
Jul 17 18:41:12.935043 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:12.935005    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7s6sp" event={"ID":"458e7dc8-c45e-42ed-abc7-b474cbbbcec5","Type":"ContainerStarted","Data":"7b46f2d9425e50b0ba8ee8a67672c75f002943f938aa4d8559b49258c3da2654"}
Jul 17 18:41:13.947052 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:13.947012    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7s6sp" event={"ID":"458e7dc8-c45e-42ed-abc7-b474cbbbcec5","Type":"ContainerStarted","Data":"b94c7c6d8ef1ce1bba11d3e4ad49c7cec16748f59410db054399b033ff6b06f7"}
Jul 17 18:41:13.983779 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:13.983727    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7s6sp" podStartSLOduration=1.473206526 podStartE2EDuration="2.983711235s" totalImagesPullingTime="1.510504709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:41:13.981095857 +0000 UTC m=+484.422604547" watchObservedRunningTime="2026-07-17 18:41:13.983711235 +0000 UTC m=+484.425219926"
Jul 17 18:41:15.334854 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:15.334819    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn"]
Jul 17 18:41:15.337412 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:15.337390    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn"
Jul 17 18:41:15.340410 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:15.340381    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 17 18:41:15.340525 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:15.340483    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r2ztx\""
Jul 17 18:41:15.341346 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:15.341329    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 17 18:41:15.355356 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:15.355334    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn"]
Jul 17 18:41:15.463268 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:15.463240    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l6th\" (UniqueName: \"kubernetes.io/projected/23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1-kube-api-access-2l6th\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn\" (UID: \"23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn"
Jul 17 18:41:15.463450 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:15.463282    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn\" (UID: \"23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn"
Jul 17 18:41:15.463450 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:15.463331    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn\" (UID: \"23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn"
Jul 17 18:41:15.564110 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:15.564073    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l6th\" (UniqueName: \"kubernetes.io/projected/23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1-kube-api-access-2l6th\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn\" (UID: \"23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn"
Jul 17 18:41:15.564110 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:15.564114    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn\" (UID: \"23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn"
Jul 17 18:41:15.564366 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:15.564139    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn\" (UID: \"23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn"
Jul 17 18:41:15.564514 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:15.564491    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn\" (UID: \"23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn"
Jul 17 18:41:15.564549 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:15.564527    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn\" (UID: \"23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn"
Jul 17 18:41:15.572826 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:15.572807    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l6th\" (UniqueName: \"kubernetes.io/projected/23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1-kube-api-access-2l6th\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn\" (UID: \"23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn"
Jul 17 18:41:15.646868 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:15.646805    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn"
Jul 17 18:41:15.798497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:15.798464    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn"]
Jul 17 18:41:15.800120 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:41:15.800091    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c65dc0_0cf7_43d0_9d3b_ad933ebe4ae1.slice/crio-738f7d7f82d5f8a9db89d3a8b9fe3c792e20dbc68812e19f2196f052f720edbc WatchSource:0}: Error finding container 738f7d7f82d5f8a9db89d3a8b9fe3c792e20dbc68812e19f2196f052f720edbc: Status 404 returned error can't find the container with id 738f7d7f82d5f8a9db89d3a8b9fe3c792e20dbc68812e19f2196f052f720edbc
Jul 17 18:41:15.956875 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:15.956792    2559 generic.go:356] "Generic (PLEG): container finished" podID="23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1" containerID="846a9ce2cbbe3c1dc47537c5dc46fc798dcb064146f2ca399ae64996b99f99a5" exitCode=0
Jul 17 18:41:15.956875 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:15.956837    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn" event={"ID":"23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1","Type":"ContainerDied","Data":"846a9ce2cbbe3c1dc47537c5dc46fc798dcb064146f2ca399ae64996b99f99a5"}
Jul 17 18:41:15.956875 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:15.956872    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn" event={"ID":"23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1","Type":"ContainerStarted","Data":"738f7d7f82d5f8a9db89d3a8b9fe3c792e20dbc68812e19f2196f052f720edbc"}
Jul 17 18:41:21.984415 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:21.984376    2559 generic.go:356] "Generic (PLEG): container finished" podID="23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1" containerID="f0e2946c07d657024774dee40fb1c42af09aa0c9e14847a0f7c8bb354b04679a" exitCode=0
Jul 17 18:41:21.984847 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:21.984454    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn" event={"ID":"23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1","Type":"ContainerDied","Data":"f0e2946c07d657024774dee40fb1c42af09aa0c9e14847a0f7c8bb354b04679a"}
Jul 17 18:41:22.990807 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:22.990771    2559 generic.go:356] "Generic (PLEG): container finished" podID="23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1" containerID="a7926ab7d327bd9ccba998e4f43b7eec3dcad1eda8c233f0ff4a73fad6c5442f" exitCode=0
Jul 17 18:41:22.991215 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:22.990830    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn" event={"ID":"23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1","Type":"ContainerDied","Data":"a7926ab7d327bd9ccba998e4f43b7eec3dcad1eda8c233f0ff4a73fad6c5442f"}
Jul 17 18:41:23.391529 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:23.391500    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7vqkt"]
Jul 17 18:41:23.393781 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:23.393766    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7vqkt"
Jul 17 18:41:23.396438 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:23.396414    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-w8td6\""
Jul 17 18:41:23.396553 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:23.396433    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 17 18:41:23.396695 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:23.396682    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 17 18:41:23.404330 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:23.404281    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7vqkt"]
Jul 17 18:41:23.533614 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:23.533581    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ae21bba-9262-4646-9f40-8aa79e721346-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7vqkt\" (UID: \"0ae21bba-9262-4646-9f40-8aa79e721346\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7vqkt"
Jul 17 18:41:23.533773 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:23.533631    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ds6f\" (UniqueName: \"kubernetes.io/projected/0ae21bba-9262-4646-9f40-8aa79e721346-kube-api-access-5ds6f\") pod \"cert-manager-cainjector-559cdd8c87-7vqkt\" (UID: \"0ae21bba-9262-4646-9f40-8aa79e721346\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7vqkt"
Jul 17 18:41:23.634818 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:23.634792    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ds6f\" (UniqueName: \"kubernetes.io/projected/0ae21bba-9262-4646-9f40-8aa79e721346-kube-api-access-5ds6f\") pod \"cert-manager-cainjector-559cdd8c87-7vqkt\" (UID: \"0ae21bba-9262-4646-9f40-8aa79e721346\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7vqkt"
Jul 17 18:41:23.634963 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:23.634845    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ae21bba-9262-4646-9f40-8aa79e721346-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7vqkt\" (UID: \"0ae21bba-9262-4646-9f40-8aa79e721346\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7vqkt"
Jul 17 18:41:23.644965 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:23.644820    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ae21bba-9262-4646-9f40-8aa79e721346-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7vqkt\" (UID: \"0ae21bba-9262-4646-9f40-8aa79e721346\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7vqkt"
Jul 17 18:41:23.646567 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:23.646548    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ds6f\" (UniqueName: \"kubernetes.io/projected/0ae21bba-9262-4646-9f40-8aa79e721346-kube-api-access-5ds6f\") pod \"cert-manager-cainjector-559cdd8c87-7vqkt\" (UID: \"0ae21bba-9262-4646-9f40-8aa79e721346\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7vqkt"
Jul 17 18:41:23.703092 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:23.703070    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7vqkt"
Jul 17 18:41:23.834732 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:23.834704    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7vqkt"]
Jul 17 18:41:23.838293 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:41:23.838266    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ae21bba_9262_4646_9f40_8aa79e721346.slice/crio-cdf8f2d8342e03b1cff52d93ecbf718e5ad70bf78a1bd964f59c91e9fe13a2fc WatchSource:0}: Error finding container cdf8f2d8342e03b1cff52d93ecbf718e5ad70bf78a1bd964f59c91e9fe13a2fc: Status 404 returned error can't find the container with id cdf8f2d8342e03b1cff52d93ecbf718e5ad70bf78a1bd964f59c91e9fe13a2fc
Jul 17 18:41:23.996908 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:23.996814    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7vqkt" event={"ID":"0ae21bba-9262-4646-9f40-8aa79e721346","Type":"ContainerStarted","Data":"cdf8f2d8342e03b1cff52d93ecbf718e5ad70bf78a1bd964f59c91e9fe13a2fc"}
Jul 17 18:41:24.114599 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:24.114567    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn"
Jul 17 18:41:24.239102 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:24.239067    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1-util\") pod \"23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1\" (UID: \"23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1\") "
Jul 17 18:41:24.239290 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:24.239145    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l6th\" (UniqueName: \"kubernetes.io/projected/23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1-kube-api-access-2l6th\") pod \"23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1\" (UID: \"23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1\") "
Jul 17 18:41:24.239290 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:24.239179    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1-bundle\") pod \"23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1\" (UID: \"23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1\") "
Jul 17 18:41:24.239618 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:24.239591    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1-bundle" (OuterVolumeSpecName: "bundle") pod "23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1" (UID: "23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 18:41:24.241533 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:24.241515    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1-kube-api-access-2l6th" (OuterVolumeSpecName: "kube-api-access-2l6th") pod "23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1" (UID: "23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1"). InnerVolumeSpecName "kube-api-access-2l6th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:41:24.243717 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:24.243685    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1-util" (OuterVolumeSpecName: "util") pod "23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1" (UID: "23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 18:41:24.340611 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:24.340585    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1-bundle\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:41:24.340611 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:24.340608    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1-util\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:41:24.340755 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:24.340617    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l6th\" (UniqueName: \"kubernetes.io/projected/23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1-kube-api-access-2l6th\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:41:25.003725 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:25.003643    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn"
Jul 17 18:41:25.003725 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:25.003661    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsdvjn" event={"ID":"23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1","Type":"ContainerDied","Data":"738f7d7f82d5f8a9db89d3a8b9fe3c792e20dbc68812e19f2196f052f720edbc"}
Jul 17 18:41:25.003725 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:25.003714    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="738f7d7f82d5f8a9db89d3a8b9fe3c792e20dbc68812e19f2196f052f720edbc"
Jul 17 18:41:26.497061 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:26.497024    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4hkhf"]
Jul 17 18:41:26.497496 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:26.497362    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1" containerName="util"
Jul 17 18:41:26.497496 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:26.497375    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1" containerName="util"
Jul 17 18:41:26.497496 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:26.497385    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1" containerName="pull"
Jul 17 18:41:26.497496 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:26.497390    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1" containerName="pull"
Jul 17 18:41:26.497496 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:26.497397    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1" containerName="extract"
Jul 17 18:41:26.497496 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:26.497403    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1" containerName="extract"
Jul 17 18:41:26.497496 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:26.497466    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="23c65dc0-0cf7-43d0-9d3b-ad933ebe4ae1" containerName="extract"
Jul 17 18:41:26.499510 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:26.499495    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4hkhf"
Jul 17 18:41:26.503038 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:26.503017    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-cpndj\""
Jul 17 18:41:26.519112 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:26.519085    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4hkhf"]
Jul 17 18:41:26.661500 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:26.661402    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49lvt\" (UniqueName: \"kubernetes.io/projected/6f15d307-dcd9-4d21-a09a-731c4245a17a-kube-api-access-49lvt\") pod \"cert-manager-55b4d954d5-4hkhf\" (UID: \"6f15d307-dcd9-4d21-a09a-731c4245a17a\") " pod="cert-manager/cert-manager-55b4d954d5-4hkhf"
Jul 17 18:41:26.661500 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:26.661473    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f15d307-dcd9-4d21-a09a-731c4245a17a-bound-sa-token\") pod \"cert-manager-55b4d954d5-4hkhf\" (UID: \"6f15d307-dcd9-4d21-a09a-731c4245a17a\") " pod="cert-manager/cert-manager-55b4d954d5-4hkhf"
Jul 17 18:41:26.762365 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:26.762298    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49lvt\" (UniqueName: \"kubernetes.io/projected/6f15d307-dcd9-4d21-a09a-731c4245a17a-kube-api-access-49lvt\") pod \"cert-manager-55b4d954d5-4hkhf\" (UID: \"6f15d307-dcd9-4d21-a09a-731c4245a17a\") " pod="cert-manager/cert-manager-55b4d954d5-4hkhf"
Jul 17 18:41:26.762533 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:26.762404    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f15d307-dcd9-4d21-a09a-731c4245a17a-bound-sa-token\") pod \"cert-manager-55b4d954d5-4hkhf\" (UID: \"6f15d307-dcd9-4d21-a09a-731c4245a17a\") " pod="cert-manager/cert-manager-55b4d954d5-4hkhf"
Jul 17 18:41:26.771583 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:26.771543    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49lvt\" (UniqueName: \"kubernetes.io/projected/6f15d307-dcd9-4d21-a09a-731c4245a17a-kube-api-access-49lvt\") pod \"cert-manager-55b4d954d5-4hkhf\" (UID: \"6f15d307-dcd9-4d21-a09a-731c4245a17a\") " pod="cert-manager/cert-manager-55b4d954d5-4hkhf"
Jul 17 18:41:26.771717 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:26.771546    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f15d307-dcd9-4d21-a09a-731c4245a17a-bound-sa-token\") pod \"cert-manager-55b4d954d5-4hkhf\" (UID: \"6f15d307-dcd9-4d21-a09a-731c4245a17a\") " pod="cert-manager/cert-manager-55b4d954d5-4hkhf"
Jul 17 18:41:26.810164 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:26.810130    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4hkhf"
Jul 17 18:41:26.956750 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:26.956728    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4hkhf"]
Jul 17 18:41:26.959036 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:41:26.959013    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f15d307_dcd9_4d21_a09a_731c4245a17a.slice/crio-c7304733a700e1a1cd91b54809f68ec5abc97cc71a00154af8a587301da8f28f WatchSource:0}: Error finding container c7304733a700e1a1cd91b54809f68ec5abc97cc71a00154af8a587301da8f28f: Status 404 returned error can't find the container with id c7304733a700e1a1cd91b54809f68ec5abc97cc71a00154af8a587301da8f28f
Jul 17 18:41:27.015374 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:27.015339    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7vqkt" event={"ID":"0ae21bba-9262-4646-9f40-8aa79e721346","Type":"ContainerStarted","Data":"52cb1a55c5209c92c18e3b8c49ee493d2ae9c0c07c7272cd04037b7bd0c38021"}
Jul 17 18:41:27.016987 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:27.016963    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4hkhf" event={"ID":"6f15d307-dcd9-4d21-a09a-731c4245a17a","Type":"ContainerStarted","Data":"c7304733a700e1a1cd91b54809f68ec5abc97cc71a00154af8a587301da8f28f"}
Jul 17 18:41:27.037767 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:27.037726    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7vqkt" podStartSLOduration=1.525905002 podStartE2EDuration="4.037709705s" totalImagesPullingTime="2.511804703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:41:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:41:27.035622082 +0000 UTC m=+497.477130773" watchObservedRunningTime="2026-07-17 18:41:27.037709705 +0000 UTC m=+497.479218399"
Jul 17 18:41:27.057888 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:27.057829    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-4hkhf" podStartSLOduration=1.057805668 podStartE2EDuration="1.057805668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:41:27.05568207 +0000 UTC m=+497.497190761" watchObservedRunningTime="2026-07-17 18:41:27.057805668 +0000 UTC m=+497.499314364"
Jul 17 18:41:28.023378 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:28.023338    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4hkhf" event={"ID":"6f15d307-dcd9-4d21-a09a-731c4245a17a","Type":"ContainerStarted","Data":"02baa93fd69c84284ce1e210cb070ce3b3c68dc80df201f32cb1171b8d2a840e"}
Jul 17 18:41:34.668147 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:34.668114    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg"]
Jul 17 18:41:34.670512 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:34.670496    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg"
Jul 17 18:41:34.673196 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:34.673162    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 17 18:41:34.673352 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:34.673268    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 17 18:41:34.674370 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:34.674352    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r2ztx\""
Jul 17 18:41:34.680194 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:34.680176    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg"]
Jul 17 18:41:34.722683 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:34.722660    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74239c59-613e-4ebb-9b3b-9f276bfdcbc1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg\" (UID: \"74239c59-613e-4ebb-9b3b-9f276bfdcbc1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg"
Jul 17 18:41:34.722780 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:34.722717    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b244\" (UniqueName: \"kubernetes.io/projected/74239c59-613e-4ebb-9b3b-9f276bfdcbc1-kube-api-access-2b244\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg\" (UID: \"74239c59-613e-4ebb-9b3b-9f276bfdcbc1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg"
Jul 17 18:41:34.722780 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:34.722744    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74239c59-613e-4ebb-9b3b-9f276bfdcbc1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg\" (UID: \"74239c59-613e-4ebb-9b3b-9f276bfdcbc1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg"
Jul 17 18:41:34.823435 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:34.823407    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74239c59-613e-4ebb-9b3b-9f276bfdcbc1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg\" (UID: \"74239c59-613e-4ebb-9b3b-9f276bfdcbc1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg"
Jul 17 18:41:34.823530 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:34.823483    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b244\" (UniqueName: \"kubernetes.io/projected/74239c59-613e-4ebb-9b3b-9f276bfdcbc1-kube-api-access-2b244\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg\" (UID: \"74239c59-613e-4ebb-9b3b-9f276bfdcbc1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg"
Jul 17 18:41:34.823530 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:34.823516    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74239c59-613e-4ebb-9b3b-9f276bfdcbc1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg\" (UID: \"74239c59-613e-4ebb-9b3b-9f276bfdcbc1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg"
Jul 17 18:41:34.823823 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:34.823804    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74239c59-613e-4ebb-9b3b-9f276bfdcbc1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg\" (UID: \"74239c59-613e-4ebb-9b3b-9f276bfdcbc1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg"
Jul 17 18:41:34.823860 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:34.823820    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74239c59-613e-4ebb-9b3b-9f276bfdcbc1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg\" (UID: \"74239c59-613e-4ebb-9b3b-9f276bfdcbc1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg"
Jul 17 18:41:34.832097 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:34.832080    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b244\" (UniqueName: \"kubernetes.io/projected/74239c59-613e-4ebb-9b3b-9f276bfdcbc1-kube-api-access-2b244\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg\" (UID: \"74239c59-613e-4ebb-9b3b-9f276bfdcbc1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg"
Jul 17 18:41:34.980066 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:34.980013    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg"
Jul 17 18:41:35.114630 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:35.114605    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg"]
Jul 17 18:41:35.116675 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:41:35.116647    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74239c59_613e_4ebb_9b3b_9f276bfdcbc1.slice/crio-64b2f1c9b8947ab54101f96387a7cee479e325d20dbca3ce3709f531f23ea3c9 WatchSource:0}: Error finding container 64b2f1c9b8947ab54101f96387a7cee479e325d20dbca3ce3709f531f23ea3c9: Status 404 returned error can't find the container with id 64b2f1c9b8947ab54101f96387a7cee479e325d20dbca3ce3709f531f23ea3c9
Jul 17 18:41:36.060667 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:36.060636    2559 generic.go:356] "Generic (PLEG): container finished" podID="74239c59-613e-4ebb-9b3b-9f276bfdcbc1" containerID="b039785b048c82e9adc7b05bafbb582d26bc79c6c2c5a0d89855e45c66492e26" exitCode=0
Jul 17 18:41:36.061211 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:36.060723    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg" event={"ID":"74239c59-613e-4ebb-9b3b-9f276bfdcbc1","Type":"ContainerDied","Data":"b039785b048c82e9adc7b05bafbb582d26bc79c6c2c5a0d89855e45c66492e26"}
Jul 17 18:41:36.061211 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:36.060772    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg" event={"ID":"74239c59-613e-4ebb-9b3b-9f276bfdcbc1","Type":"ContainerStarted","Data":"64b2f1c9b8947ab54101f96387a7cee479e325d20dbca3ce3709f531f23ea3c9"}
Jul 17 18:41:37.067141 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:37.067108    2559 generic.go:356] "Generic (PLEG): container finished" podID="74239c59-613e-4ebb-9b3b-9f276bfdcbc1" containerID="99b2a3da3d3dbb0aa8f3295c9c9ba85210dd67965d9212f1895ce3505c96c29a" exitCode=0
Jul 17 18:41:37.067540 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:37.067191    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg" event={"ID":"74239c59-613e-4ebb-9b3b-9f276bfdcbc1","Type":"ContainerDied","Data":"99b2a3da3d3dbb0aa8f3295c9c9ba85210dd67965d9212f1895ce3505c96c29a"}
Jul 17 18:41:38.074115 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:38.074080    2559 generic.go:356] "Generic (PLEG): container finished" podID="74239c59-613e-4ebb-9b3b-9f276bfdcbc1" containerID="cd3139289459e3059afebffa1c62d417f86dc4cec754b6f920622235a596d525" exitCode=0
Jul 17 18:41:38.074505 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:38.074144    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg" event={"ID":"74239c59-613e-4ebb-9b3b-9f276bfdcbc1","Type":"ContainerDied","Data":"cd3139289459e3059afebffa1c62d417f86dc4cec754b6f920622235a596d525"}
Jul 17 18:41:39.202324 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:39.202284    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg"
Jul 17 18:41:39.256424 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:39.256400    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b244\" (UniqueName: \"kubernetes.io/projected/74239c59-613e-4ebb-9b3b-9f276bfdcbc1-kube-api-access-2b244\") pod \"74239c59-613e-4ebb-9b3b-9f276bfdcbc1\" (UID: \"74239c59-613e-4ebb-9b3b-9f276bfdcbc1\") "
Jul 17 18:41:39.256554 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:39.256464    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74239c59-613e-4ebb-9b3b-9f276bfdcbc1-util\") pod \"74239c59-613e-4ebb-9b3b-9f276bfdcbc1\" (UID: \"74239c59-613e-4ebb-9b3b-9f276bfdcbc1\") "
Jul 17 18:41:39.256554 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:39.256497    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74239c59-613e-4ebb-9b3b-9f276bfdcbc1-bundle\") pod \"74239c59-613e-4ebb-9b3b-9f276bfdcbc1\" (UID: \"74239c59-613e-4ebb-9b3b-9f276bfdcbc1\") "
Jul 17 18:41:39.257144 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:39.257122    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74239c59-613e-4ebb-9b3b-9f276bfdcbc1-bundle" (OuterVolumeSpecName: "bundle") pod "74239c59-613e-4ebb-9b3b-9f276bfdcbc1" (UID: "74239c59-613e-4ebb-9b3b-9f276bfdcbc1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 18:41:39.258354 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:39.258334    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74239c59-613e-4ebb-9b3b-9f276bfdcbc1-kube-api-access-2b244" (OuterVolumeSpecName: "kube-api-access-2b244") pod "74239c59-613e-4ebb-9b3b-9f276bfdcbc1" (UID: "74239c59-613e-4ebb-9b3b-9f276bfdcbc1"). InnerVolumeSpecName "kube-api-access-2b244". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:41:39.265083 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:39.265044    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74239c59-613e-4ebb-9b3b-9f276bfdcbc1-util" (OuterVolumeSpecName: "util") pod "74239c59-613e-4ebb-9b3b-9f276bfdcbc1" (UID: "74239c59-613e-4ebb-9b3b-9f276bfdcbc1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 18:41:39.358057 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:39.357979    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b244\" (UniqueName: \"kubernetes.io/projected/74239c59-613e-4ebb-9b3b-9f276bfdcbc1-kube-api-access-2b244\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:41:39.358057 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:39.358004    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74239c59-613e-4ebb-9b3b-9f276bfdcbc1-util\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:41:39.358057 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:39.358015    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74239c59-613e-4ebb-9b3b-9f276bfdcbc1-bundle\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:41:40.084484 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:40.084450    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg"
Jul 17 18:41:40.084484 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:40.084460    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxktbg" event={"ID":"74239c59-613e-4ebb-9b3b-9f276bfdcbc1","Type":"ContainerDied","Data":"64b2f1c9b8947ab54101f96387a7cee479e325d20dbca3ce3709f531f23ea3c9"}
Jul 17 18:41:40.084682 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:40.084508    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64b2f1c9b8947ab54101f96387a7cee479e325d20dbca3ce3709f531f23ea3c9"
Jul 17 18:41:48.257298 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.257266    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"]
Jul 17 18:41:48.257673 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.257573    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74239c59-613e-4ebb-9b3b-9f276bfdcbc1" containerName="util"
Jul 17 18:41:48.257673 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.257583    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="74239c59-613e-4ebb-9b3b-9f276bfdcbc1" containerName="util"
Jul 17 18:41:48.257673 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.257600    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74239c59-613e-4ebb-9b3b-9f276bfdcbc1" containerName="pull"
Jul 17 18:41:48.257673 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.257605    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="74239c59-613e-4ebb-9b3b-9f276bfdcbc1" containerName="pull"
Jul 17 18:41:48.257673 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.257617    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74239c59-613e-4ebb-9b3b-9f276bfdcbc1" containerName="extract"
Jul 17 18:41:48.257673 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.257623    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="74239c59-613e-4ebb-9b3b-9f276bfdcbc1" containerName="extract"
Jul 17 18:41:48.257854 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.257681    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="74239c59-613e-4ebb-9b3b-9f276bfdcbc1" containerName="extract"
Jul 17 18:41:48.259889 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.259873    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"
Jul 17 18:41:48.264887 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.264863    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 17 18:41:48.265005 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.264939    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 17 18:41:48.265234 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.265214    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 17 18:41:48.265827 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.265804    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 17 18:41:48.266083 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.266068    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-n2nnj\""
Jul 17 18:41:48.295520 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.295499    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"]
Jul 17 18:41:48.328515 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.328489    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/101e4b60-dfa6-48e7-97e1-0de5c935dbef-manifests\") pod \"opendatahub-operator-controller-manager-57479f7bfb-xfmhk\" (UID: \"101e4b60-dfa6-48e7-97e1-0de5c935dbef\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"
Jul 17 18:41:48.328617 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.328530    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/101e4b60-dfa6-48e7-97e1-0de5c935dbef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57479f7bfb-xfmhk\" (UID: \"101e4b60-dfa6-48e7-97e1-0de5c935dbef\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"
Jul 17 18:41:48.328617 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.328561    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/101e4b60-dfa6-48e7-97e1-0de5c935dbef-webhook-cert\") pod \"opendatahub-operator-controller-manager-57479f7bfb-xfmhk\" (UID: \"101e4b60-dfa6-48e7-97e1-0de5c935dbef\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"
Jul 17 18:41:48.328617 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.328584    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sngkk\" (UniqueName: \"kubernetes.io/projected/101e4b60-dfa6-48e7-97e1-0de5c935dbef-kube-api-access-sngkk\") pod \"opendatahub-operator-controller-manager-57479f7bfb-xfmhk\" (UID: \"101e4b60-dfa6-48e7-97e1-0de5c935dbef\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"
Jul 17 18:41:48.328852 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.328668    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/101e4b60-dfa6-48e7-97e1-0de5c935dbef-tmp\") pod \"opendatahub-operator-controller-manager-57479f7bfb-xfmhk\" (UID: \"101e4b60-dfa6-48e7-97e1-0de5c935dbef\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"
Jul 17 18:41:48.429630 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.429602    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/101e4b60-dfa6-48e7-97e1-0de5c935dbef-webhook-cert\") pod \"opendatahub-operator-controller-manager-57479f7bfb-xfmhk\" (UID: \"101e4b60-dfa6-48e7-97e1-0de5c935dbef\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"
Jul 17 18:41:48.429744 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.429638    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sngkk\" (UniqueName: \"kubernetes.io/projected/101e4b60-dfa6-48e7-97e1-0de5c935dbef-kube-api-access-sngkk\") pod \"opendatahub-operator-controller-manager-57479f7bfb-xfmhk\" (UID: \"101e4b60-dfa6-48e7-97e1-0de5c935dbef\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"
Jul 17 18:41:48.429744 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.429665    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/101e4b60-dfa6-48e7-97e1-0de5c935dbef-tmp\") pod \"opendatahub-operator-controller-manager-57479f7bfb-xfmhk\" (UID: \"101e4b60-dfa6-48e7-97e1-0de5c935dbef\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"
Jul 17 18:41:48.429744 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.429703    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/101e4b60-dfa6-48e7-97e1-0de5c935dbef-manifests\") pod \"opendatahub-operator-controller-manager-57479f7bfb-xfmhk\" (UID: \"101e4b60-dfa6-48e7-97e1-0de5c935dbef\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"
Jul 17 18:41:48.429744 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.429718    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/101e4b60-dfa6-48e7-97e1-0de5c935dbef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57479f7bfb-xfmhk\" (UID: \"101e4b60-dfa6-48e7-97e1-0de5c935dbef\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"
Jul 17 18:41:48.430162 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.430133    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/101e4b60-dfa6-48e7-97e1-0de5c935dbef-manifests\") pod \"opendatahub-operator-controller-manager-57479f7bfb-xfmhk\" (UID: \"101e4b60-dfa6-48e7-97e1-0de5c935dbef\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"
Jul 17 18:41:48.432182 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.432150    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/101e4b60-dfa6-48e7-97e1-0de5c935dbef-webhook-cert\") pod \"opendatahub-operator-controller-manager-57479f7bfb-xfmhk\" (UID: \"101e4b60-dfa6-48e7-97e1-0de5c935dbef\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"
Jul 17 18:41:48.432478 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.432456    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/101e4b60-dfa6-48e7-97e1-0de5c935dbef-tmp\") pod \"opendatahub-operator-controller-manager-57479f7bfb-xfmhk\" (UID: \"101e4b60-dfa6-48e7-97e1-0de5c935dbef\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"
Jul 17 18:41:48.432612 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.432593    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/101e4b60-dfa6-48e7-97e1-0de5c935dbef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57479f7bfb-xfmhk\" (UID: \"101e4b60-dfa6-48e7-97e1-0de5c935dbef\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"
Jul 17 18:41:48.448978 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.448956    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sngkk\" (UniqueName: \"kubernetes.io/projected/101e4b60-dfa6-48e7-97e1-0de5c935dbef-kube-api-access-sngkk\") pod \"opendatahub-operator-controller-manager-57479f7bfb-xfmhk\" (UID: \"101e4b60-dfa6-48e7-97e1-0de5c935dbef\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"
Jul 17 18:41:48.570390 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.570358    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"
Jul 17 18:41:48.718287 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:48.718266    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"]
Jul 17 18:41:48.721119 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:41:48.721092    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101e4b60_dfa6_48e7_97e1_0de5c935dbef.slice/crio-1843ca63dd8c2bb25905dafdfe0ec6c6d4d67491b312712b5b60f1fc4b27b244 WatchSource:0}: Error finding container 1843ca63dd8c2bb25905dafdfe0ec6c6d4d67491b312712b5b60f1fc4b27b244: Status 404 returned error can't find the container with id 1843ca63dd8c2bb25905dafdfe0ec6c6d4d67491b312712b5b60f1fc4b27b244
Jul 17 18:41:49.125986 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:49.125954    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk" event={"ID":"101e4b60-dfa6-48e7-97e1-0de5c935dbef","Type":"ContainerStarted","Data":"1843ca63dd8c2bb25905dafdfe0ec6c6d4d67491b312712b5b60f1fc4b27b244"}
Jul 17 18:41:52.143557 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:52.143517    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk" event={"ID":"101e4b60-dfa6-48e7-97e1-0de5c935dbef","Type":"ContainerStarted","Data":"8adb6f888780587dc7c5344d841f21c16c2e8dc6ea72299189084c5ce3e7a106"}
Jul 17 18:41:53.149598 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:53.149513    2559 generic.go:356] "Generic (PLEG): container finished" podID="101e4b60-dfa6-48e7-97e1-0de5c935dbef" containerID="8adb6f888780587dc7c5344d841f21c16c2e8dc6ea72299189084c5ce3e7a106" exitCode=0
Jul 17 18:41:53.149958 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:53.149599    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk" event={"ID":"101e4b60-dfa6-48e7-97e1-0de5c935dbef","Type":"ContainerDied","Data":"8adb6f888780587dc7c5344d841f21c16c2e8dc6ea72299189084c5ce3e7a106"}
Jul 17 18:41:54.157282 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:54.157244    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk" event={"ID":"101e4b60-dfa6-48e7-97e1-0de5c935dbef","Type":"ContainerStarted","Data":"8a3cfdb18adddc1d11ec5af98622080db65d098622d8a232bf134b12b806f9cb"}
Jul 17 18:41:54.157750 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:54.157495    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"
Jul 17 18:41:54.190499 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:41:54.190448    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk" podStartSLOduration=1.804615993 podStartE2EDuration="6.190432481s" totalImagesPullingTime="3.32148131s" totalInitContainerRuntime="1.064335178s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:41:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:41:54.188935445 +0000 UTC m=+524.630444133" watchObservedRunningTime="2026-07-17 18:41:54.190432481 +0000 UTC m=+524.631941170"
Jul 17 18:42:05.164694 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:05.164617    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-xfmhk"
Jul 17 18:42:08.411133 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.411098    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-864488b8fd-fxtq4"]
Jul 17 18:42:08.414263 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.414240    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-864488b8fd-fxtq4"
Jul 17 18:42:08.420411 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.420385    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 17 18:42:08.420546 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.420441    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 17 18:42:08.420772 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.420750    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 17 18:42:08.421573 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.421540    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-85j2c\""
Jul 17 18:42:08.421686 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.421624    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 17 18:42:08.429215 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.429183    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-864488b8fd-fxtq4"]
Jul 17 18:42:08.502402 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.502368    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48x57\" (UniqueName: \"kubernetes.io/projected/367eed95-dbb9-47d7-aa40-bc764b363849-kube-api-access-48x57\") pod \"kube-auth-proxy-864488b8fd-fxtq4\" (UID: \"367eed95-dbb9-47d7-aa40-bc764b363849\") " pod="openshift-ingress/kube-auth-proxy-864488b8fd-fxtq4"
Jul 17 18:42:08.502564 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.502463    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/367eed95-dbb9-47d7-aa40-bc764b363849-tls-certs\") pod \"kube-auth-proxy-864488b8fd-fxtq4\" (UID: \"367eed95-dbb9-47d7-aa40-bc764b363849\") " pod="openshift-ingress/kube-auth-proxy-864488b8fd-fxtq4"
Jul 17 18:42:08.502564 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.502512    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/367eed95-dbb9-47d7-aa40-bc764b363849-tmp\") pod \"kube-auth-proxy-864488b8fd-fxtq4\" (UID: \"367eed95-dbb9-47d7-aa40-bc764b363849\") " pod="openshift-ingress/kube-auth-proxy-864488b8fd-fxtq4"
Jul 17 18:42:08.521270 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.521237    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95"]
Jul 17 18:42:08.528159 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.528129    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95"
Jul 17 18:42:08.532708 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.532500    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mkcfh\""
Jul 17 18:42:08.541901 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.539822    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 17 18:42:08.541901 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.540043    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 17 18:42:08.541901 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.540200    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 17 18:42:08.541901 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.540437    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 17 18:42:08.541901 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.540602    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 17 18:42:08.543217 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.543193    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95"]
Jul 17 18:42:08.603376 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.603324    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/194e96f5-01d6-45b5-b20d-a8423562d81b-cert\") pod \"lws-controller-manager-cbc4544f6-8wd95\" (UID: \"194e96f5-01d6-45b5-b20d-a8423562d81b\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95"
Jul 17 18:42:08.603588 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.603418    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48x57\" (UniqueName: \"kubernetes.io/projected/367eed95-dbb9-47d7-aa40-bc764b363849-kube-api-access-48x57\") pod \"kube-auth-proxy-864488b8fd-fxtq4\" (UID: \"367eed95-dbb9-47d7-aa40-bc764b363849\") " pod="openshift-ingress/kube-auth-proxy-864488b8fd-fxtq4"
Jul 17 18:42:08.603588 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.603467    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95k5b\" (UniqueName: \"kubernetes.io/projected/194e96f5-01d6-45b5-b20d-a8423562d81b-kube-api-access-95k5b\") pod \"lws-controller-manager-cbc4544f6-8wd95\" (UID: \"194e96f5-01d6-45b5-b20d-a8423562d81b\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95"
Jul 17 18:42:08.603588 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.603493    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/194e96f5-01d6-45b5-b20d-a8423562d81b-manager-config\") pod \"lws-controller-manager-cbc4544f6-8wd95\" (UID: \"194e96f5-01d6-45b5-b20d-a8423562d81b\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95"
Jul 17 18:42:08.603588 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.603584    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/194e96f5-01d6-45b5-b20d-a8423562d81b-metrics-cert\") pod \"lws-controller-manager-cbc4544f6-8wd95\" (UID: \"194e96f5-01d6-45b5-b20d-a8423562d81b\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95"
Jul 17 18:42:08.603880 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.603615    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/367eed95-dbb9-47d7-aa40-bc764b363849-tls-certs\") pod \"kube-auth-proxy-864488b8fd-fxtq4\" (UID: \"367eed95-dbb9-47d7-aa40-bc764b363849\") " pod="openshift-ingress/kube-auth-proxy-864488b8fd-fxtq4"
Jul 17 18:42:08.603880 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.603645    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/367eed95-dbb9-47d7-aa40-bc764b363849-tmp\") pod \"kube-auth-proxy-864488b8fd-fxtq4\" (UID: \"367eed95-dbb9-47d7-aa40-bc764b363849\") " pod="openshift-ingress/kube-auth-proxy-864488b8fd-fxtq4"
Jul 17 18:42:08.606383 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.606357    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/367eed95-dbb9-47d7-aa40-bc764b363849-tmp\") pod \"kube-auth-proxy-864488b8fd-fxtq4\" (UID: \"367eed95-dbb9-47d7-aa40-bc764b363849\") " pod="openshift-ingress/kube-auth-proxy-864488b8fd-fxtq4"
Jul 17 18:42:08.606816 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.606795    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/367eed95-dbb9-47d7-aa40-bc764b363849-tls-certs\") pod \"kube-auth-proxy-864488b8fd-fxtq4\" (UID: \"367eed95-dbb9-47d7-aa40-bc764b363849\") " pod="openshift-ingress/kube-auth-proxy-864488b8fd-fxtq4"
Jul 17 18:42:08.615590 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.615560    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48x57\" (UniqueName: \"kubernetes.io/projected/367eed95-dbb9-47d7-aa40-bc764b363849-kube-api-access-48x57\") pod \"kube-auth-proxy-864488b8fd-fxtq4\" (UID: \"367eed95-dbb9-47d7-aa40-bc764b363849\") " pod="openshift-ingress/kube-auth-proxy-864488b8fd-fxtq4"
Jul 17 18:42:08.704651 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.704560    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95k5b\" (UniqueName: \"kubernetes.io/projected/194e96f5-01d6-45b5-b20d-a8423562d81b-kube-api-access-95k5b\") pod \"lws-controller-manager-cbc4544f6-8wd95\" (UID: \"194e96f5-01d6-45b5-b20d-a8423562d81b\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95"
Jul 17 18:42:08.704651 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.704596    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/194e96f5-01d6-45b5-b20d-a8423562d81b-manager-config\") pod \"lws-controller-manager-cbc4544f6-8wd95\" (UID: \"194e96f5-01d6-45b5-b20d-a8423562d81b\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95"
Jul 17 18:42:08.704651 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.704639    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/194e96f5-01d6-45b5-b20d-a8423562d81b-metrics-cert\") pod \"lws-controller-manager-cbc4544f6-8wd95\" (UID: \"194e96f5-01d6-45b5-b20d-a8423562d81b\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95"
Jul 17 18:42:08.704885 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.704671    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/194e96f5-01d6-45b5-b20d-a8423562d81b-cert\") pod \"lws-controller-manager-cbc4544f6-8wd95\" (UID: \"194e96f5-01d6-45b5-b20d-a8423562d81b\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95"
Jul 17 18:42:08.705445 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.705389    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/194e96f5-01d6-45b5-b20d-a8423562d81b-manager-config\") pod \"lws-controller-manager-cbc4544f6-8wd95\" (UID: \"194e96f5-01d6-45b5-b20d-a8423562d81b\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95"
Jul 17 18:42:08.707298 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.707266    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/194e96f5-01d6-45b5-b20d-a8423562d81b-cert\") pod \"lws-controller-manager-cbc4544f6-8wd95\" (UID: \"194e96f5-01d6-45b5-b20d-a8423562d81b\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95"
Jul 17 18:42:08.707451 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.707430    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/194e96f5-01d6-45b5-b20d-a8423562d81b-metrics-cert\") pod \"lws-controller-manager-cbc4544f6-8wd95\" (UID: \"194e96f5-01d6-45b5-b20d-a8423562d81b\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95"
Jul 17 18:42:08.721563 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.721540    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95k5b\" (UniqueName: \"kubernetes.io/projected/194e96f5-01d6-45b5-b20d-a8423562d81b-kube-api-access-95k5b\") pod \"lws-controller-manager-cbc4544f6-8wd95\" (UID: \"194e96f5-01d6-45b5-b20d-a8423562d81b\") " pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95"
Jul 17 18:42:08.724389 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.724370    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-864488b8fd-fxtq4"
Jul 17 18:42:08.842752 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.842725    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95"
Jul 17 18:42:08.920276 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.920244    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-864488b8fd-fxtq4"]
Jul 17 18:42:08.922234 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:42:08.922209    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367eed95_dbb9_47d7_aa40_bc764b363849.slice/crio-214b47ab8aa6835ca572f4adf96fa894824dac7510c1c893479722bd9ae4d7ba WatchSource:0}: Error finding container 214b47ab8aa6835ca572f4adf96fa894824dac7510c1c893479722bd9ae4d7ba: Status 404 returned error can't find the container with id 214b47ab8aa6835ca572f4adf96fa894824dac7510c1c893479722bd9ae4d7ba
Jul 17 18:42:08.997988 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:08.997966    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95"]
Jul 17 18:42:08.999202 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:42:08.999172    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194e96f5_01d6_45b5_b20d_a8423562d81b.slice/crio-365dc6d90a0516464cb99db3405c7c91848c9075e124d8ff0a4cdca1bd544359 WatchSource:0}: Error finding container 365dc6d90a0516464cb99db3405c7c91848c9075e124d8ff0a4cdca1bd544359: Status 404 returned error can't find the container with id 365dc6d90a0516464cb99db3405c7c91848c9075e124d8ff0a4cdca1bd544359
Jul 17 18:42:09.232634 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:09.232588    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-864488b8fd-fxtq4" event={"ID":"367eed95-dbb9-47d7-aa40-bc764b363849","Type":"ContainerStarted","Data":"214b47ab8aa6835ca572f4adf96fa894824dac7510c1c893479722bd9ae4d7ba"}
Jul 17 18:42:09.233922 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:09.233884    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95" event={"ID":"194e96f5-01d6-45b5-b20d-a8423562d81b","Type":"ContainerStarted","Data":"365dc6d90a0516464cb99db3405c7c91848c9075e124d8ff0a4cdca1bd544359"}
Jul 17 18:42:11.247336 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:11.247273    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95" event={"ID":"194e96f5-01d6-45b5-b20d-a8423562d81b","Type":"ContainerStarted","Data":"3280ec1aea4014418e3fd8ae2a000c6df9d606490e579391c0ed0d6a3bda8400"}
Jul 17 18:42:11.248036 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:11.248013    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95"
Jul 17 18:42:11.272372 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:11.272296    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95" podStartSLOduration=1.743057659 podStartE2EDuration="3.272276124s" totalImagesPullingTime="1.529218465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:42:11.271076914 +0000 UTC m=+541.712585622" watchObservedRunningTime="2026-07-17 18:42:11.272276124 +0000 UTC m=+541.713784792"
Jul 17 18:42:12.730995 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:12.730967    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 17 18:42:13.261072 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:13.261028    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-864488b8fd-fxtq4" event={"ID":"367eed95-dbb9-47d7-aa40-bc764b363849","Type":"ContainerStarted","Data":"40257749f3faa0f59d5efc0262691b0ae54768208ba1bf951dee65da0a1af170"}
Jul 17 18:42:13.282096 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:13.282037    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-864488b8fd-fxtq4" podStartSLOduration=1.478003335 podStartE2EDuration="5.282021353s" totalImagesPullingTime="3.804018018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:42:13.280877526 +0000 UTC m=+543.722386215" watchObservedRunningTime="2026-07-17 18:42:13.282021353 +0000 UTC m=+543.723530043"
Jul 17 18:42:23.263889 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:42:23.263856    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-cbc4544f6-8wd95"
Jul 17 18:43:10.041063 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:10.041025    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 18:43:10.041812 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:10.041794    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 18:43:10.049483 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:10.049463    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 18:43:10.049801 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:10.049784    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 18:43:24.473572 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.473533    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64"]
Jul 17 18:43:24.477085 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.477066    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64"
Jul 17 18:43:24.482614 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.482592    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 18:43:24.482709 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.482599    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 18:43:24.483547 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.483470    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tzbmh\""
Jul 17 18:43:24.492092 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.492064    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64"]
Jul 17 18:43:24.563201 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.563176    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwsk5\" (UniqueName: \"kubernetes.io/projected/6624d4e9-b35a-4073-bda3-4dbf001ba7e0-kube-api-access-dwsk5\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64\" (UID: \"6624d4e9-b35a-4073-bda3-4dbf001ba7e0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64"
Jul 17 18:43:24.563385 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.563260    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6624d4e9-b35a-4073-bda3-4dbf001ba7e0-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64\" (UID: \"6624d4e9-b35a-4073-bda3-4dbf001ba7e0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64"
Jul 17 18:43:24.563385 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.563279    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6624d4e9-b35a-4073-bda3-4dbf001ba7e0-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64\" (UID: \"6624d4e9-b35a-4073-bda3-4dbf001ba7e0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64"
Jul 17 18:43:24.664140 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.664111    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwsk5\" (UniqueName: \"kubernetes.io/projected/6624d4e9-b35a-4073-bda3-4dbf001ba7e0-kube-api-access-dwsk5\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64\" (UID: \"6624d4e9-b35a-4073-bda3-4dbf001ba7e0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64"
Jul 17 18:43:24.664288 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.664170    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6624d4e9-b35a-4073-bda3-4dbf001ba7e0-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64\" (UID: \"6624d4e9-b35a-4073-bda3-4dbf001ba7e0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64"
Jul 17 18:43:24.664288 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.664190    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6624d4e9-b35a-4073-bda3-4dbf001ba7e0-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64\" (UID: \"6624d4e9-b35a-4073-bda3-4dbf001ba7e0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64"
Jul 17 18:43:24.664552 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.664537    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6624d4e9-b35a-4073-bda3-4dbf001ba7e0-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64\" (UID: \"6624d4e9-b35a-4073-bda3-4dbf001ba7e0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64"
Jul 17 18:43:24.664594 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.664575    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6624d4e9-b35a-4073-bda3-4dbf001ba7e0-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64\" (UID: \"6624d4e9-b35a-4073-bda3-4dbf001ba7e0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64"
Jul 17 18:43:24.673903 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.673881    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwsk5\" (UniqueName: \"kubernetes.io/projected/6624d4e9-b35a-4073-bda3-4dbf001ba7e0-kube-api-access-dwsk5\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64\" (UID: \"6624d4e9-b35a-4073-bda3-4dbf001ba7e0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64"
Jul 17 18:43:24.786058 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.785988    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64"
Jul 17 18:43:24.870560 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.867457    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29"]
Jul 17 18:43:24.876937 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.876865    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29"
Jul 17 18:43:24.880154 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.880096    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29"]
Jul 17 18:43:24.930916 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.930887    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64"]
Jul 17 18:43:24.934186 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:43:24.934160    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6624d4e9_b35a_4073_bda3_4dbf001ba7e0.slice/crio-5ddfca921237945ccfb2b3d1584aad97bfce59256d2824678ff2da1af4ca625e WatchSource:0}: Error finding container 5ddfca921237945ccfb2b3d1584aad97bfce59256d2824678ff2da1af4ca625e: Status 404 returned error can't find the container with id 5ddfca921237945ccfb2b3d1584aad97bfce59256d2824678ff2da1af4ca625e
Jul 17 18:43:24.967547 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.967523    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlxjw\" (UniqueName: \"kubernetes.io/projected/bfb82736-cf62-4f65-b8ae-d7f58162fce6-kube-api-access-mlxjw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29\" (UID: \"bfb82736-cf62-4f65-b8ae-d7f58162fce6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29"
Jul 17 18:43:24.967642 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.967563    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfb82736-cf62-4f65-b8ae-d7f58162fce6-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29\" (UID: \"bfb82736-cf62-4f65-b8ae-d7f58162fce6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29"
Jul 17 18:43:24.967642 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:24.967628    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfb82736-cf62-4f65-b8ae-d7f58162fce6-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29\" (UID: \"bfb82736-cf62-4f65-b8ae-d7f58162fce6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29"
Jul 17 18:43:25.068241 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.068213    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfb82736-cf62-4f65-b8ae-d7f58162fce6-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29\" (UID: \"bfb82736-cf62-4f65-b8ae-d7f58162fce6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29"
Jul 17 18:43:25.068380 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.068255    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlxjw\" (UniqueName: \"kubernetes.io/projected/bfb82736-cf62-4f65-b8ae-d7f58162fce6-kube-api-access-mlxjw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29\" (UID: \"bfb82736-cf62-4f65-b8ae-d7f58162fce6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29"
Jul 17 18:43:25.068380 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.068288    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfb82736-cf62-4f65-b8ae-d7f58162fce6-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29\" (UID: \"bfb82736-cf62-4f65-b8ae-d7f58162fce6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29"
Jul 17 18:43:25.068588 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.068568    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfb82736-cf62-4f65-b8ae-d7f58162fce6-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29\" (UID: \"bfb82736-cf62-4f65-b8ae-d7f58162fce6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29"
Jul 17 18:43:25.068638 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.068616    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfb82736-cf62-4f65-b8ae-d7f58162fce6-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29\" (UID: \"bfb82736-cf62-4f65-b8ae-d7f58162fce6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29"
Jul 17 18:43:25.080957 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.080939    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlxjw\" (UniqueName: \"kubernetes.io/projected/bfb82736-cf62-4f65-b8ae-d7f58162fce6-kube-api-access-mlxjw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29\" (UID: \"bfb82736-cf62-4f65-b8ae-d7f58162fce6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29"
Jul 17 18:43:25.188924 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.188904    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29"
Jul 17 18:43:25.272015 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.271987    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc"]
Jul 17 18:43:25.276007 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.275987    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc"
Jul 17 18:43:25.284429 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.284405    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc"]
Jul 17 18:43:25.320817 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.320795    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29"]
Jul 17 18:43:25.322605 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:43:25.322584    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb82736_cf62_4f65_b8ae_d7f58162fce6.slice/crio-8c4c7ee5bf26a12cc76cabf64f4edc41fe4df3822fd95c11ca70746c568ecb0b WatchSource:0}: Error finding container 8c4c7ee5bf26a12cc76cabf64f4edc41fe4df3822fd95c11ca70746c568ecb0b: Status 404 returned error can't find the container with id 8c4c7ee5bf26a12cc76cabf64f4edc41fe4df3822fd95c11ca70746c568ecb0b
Jul 17 18:43:25.370245 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.370224    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1dc407dc-206b-44f2-9934-19d747560cb4-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc\" (UID: \"1dc407dc-206b-44f2-9934-19d747560cb4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc"
Jul 17 18:43:25.370343 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.370261    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1dc407dc-206b-44f2-9934-19d747560cb4-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc\" (UID: \"1dc407dc-206b-44f2-9934-19d747560cb4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc"
Jul 17 18:43:25.370343 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.370327    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md4tt\" (UniqueName: \"kubernetes.io/projected/1dc407dc-206b-44f2-9934-19d747560cb4-kube-api-access-md4tt\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc\" (UID: \"1dc407dc-206b-44f2-9934-19d747560cb4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc"
Jul 17 18:43:25.471665 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.471635    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md4tt\" (UniqueName: \"kubernetes.io/projected/1dc407dc-206b-44f2-9934-19d747560cb4-kube-api-access-md4tt\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc\" (UID: \"1dc407dc-206b-44f2-9934-19d747560cb4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc"
Jul 17 18:43:25.471796 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.471706    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1dc407dc-206b-44f2-9934-19d747560cb4-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc\" (UID: \"1dc407dc-206b-44f2-9934-19d747560cb4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc"
Jul 17 18:43:25.471796 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.471739    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1dc407dc-206b-44f2-9934-19d747560cb4-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc\" (UID: \"1dc407dc-206b-44f2-9934-19d747560cb4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc"
Jul 17 18:43:25.472047 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.472029    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1dc407dc-206b-44f2-9934-19d747560cb4-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc\" (UID: \"1dc407dc-206b-44f2-9934-19d747560cb4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc"
Jul 17 18:43:25.472103 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.472055    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1dc407dc-206b-44f2-9934-19d747560cb4-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc\" (UID: \"1dc407dc-206b-44f2-9934-19d747560cb4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc"
Jul 17 18:43:25.481552 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.481534    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md4tt\" (UniqueName: \"kubernetes.io/projected/1dc407dc-206b-44f2-9934-19d747560cb4-kube-api-access-md4tt\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc\" (UID: \"1dc407dc-206b-44f2-9934-19d747560cb4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc"
Jul 17 18:43:25.587077 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.587026    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc"
Jul 17 18:43:25.614673 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.614638    2559 generic.go:356] "Generic (PLEG): container finished" podID="bfb82736-cf62-4f65-b8ae-d7f58162fce6" containerID="7209a90f588d04d4670333ed086f1dd54ff00d94b5b1160e8eb473ed80a1b22b" exitCode=0
Jul 17 18:43:25.614821 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.614739    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29" event={"ID":"bfb82736-cf62-4f65-b8ae-d7f58162fce6","Type":"ContainerDied","Data":"7209a90f588d04d4670333ed086f1dd54ff00d94b5b1160e8eb473ed80a1b22b"}
Jul 17 18:43:25.614821 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.614799    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29" event={"ID":"bfb82736-cf62-4f65-b8ae-d7f58162fce6","Type":"ContainerStarted","Data":"8c4c7ee5bf26a12cc76cabf64f4edc41fe4df3822fd95c11ca70746c568ecb0b"}
Jul 17 18:43:25.615297 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.615278    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 18:43:25.616558 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.616539    2559 generic.go:356] "Generic (PLEG): container finished" podID="6624d4e9-b35a-4073-bda3-4dbf001ba7e0" containerID="e9713903c25a9f4347a30d00149880c2b66c2a1ff5c9818155b9058fd6ad73ca" exitCode=0
Jul 17 18:43:25.616640 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.616621    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64" event={"ID":"6624d4e9-b35a-4073-bda3-4dbf001ba7e0","Type":"ContainerDied","Data":"e9713903c25a9f4347a30d00149880c2b66c2a1ff5c9818155b9058fd6ad73ca"}
Jul 17 18:43:25.616677 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.616651    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64" event={"ID":"6624d4e9-b35a-4073-bda3-4dbf001ba7e0","Type":"ContainerStarted","Data":"5ddfca921237945ccfb2b3d1584aad97bfce59256d2824678ff2da1af4ca625e"}
Jul 17 18:43:25.732738 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.732715    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc"]
Jul 17 18:43:25.734791 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:43:25.734761    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc407dc_206b_44f2_9934_19d747560cb4.slice/crio-9a1c312163b739b507bfe1c69bd1112f0a8935749373a16df29dbdde1c1e21d9 WatchSource:0}: Error finding container 9a1c312163b739b507bfe1c69bd1112f0a8935749373a16df29dbdde1c1e21d9: Status 404 returned error can't find the container with id 9a1c312163b739b507bfe1c69bd1112f0a8935749373a16df29dbdde1c1e21d9
Jul 17 18:43:25.738359 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.738337    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds"]
Jul 17 18:43:25.742658 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.742642    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds"
Jul 17 18:43:25.768571 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.768545    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds"]
Jul 17 18:43:25.774975 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.774945    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds\" (UID: \"d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds"
Jul 17 18:43:25.775088 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.774992    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds\" (UID: \"d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds"
Jul 17 18:43:25.775151 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.775118    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlcsc\" (UniqueName: \"kubernetes.io/projected/d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5-kube-api-access-tlcsc\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds\" (UID: \"d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds"
Jul 17 18:43:25.875814 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.875786    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlcsc\" (UniqueName: \"kubernetes.io/projected/d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5-kube-api-access-tlcsc\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds\" (UID: \"d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds"
Jul 17 18:43:25.875940 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.875846    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds\" (UID: \"d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds"
Jul 17 18:43:25.875940 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.875867    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds\" (UID: \"d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds"
Jul 17 18:43:25.876178 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.876164    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds\" (UID: \"d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds"
Jul 17 18:43:25.876241 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.876222    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds\" (UID: \"d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds"
Jul 17 18:43:25.894260 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:25.894236    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlcsc\" (UniqueName: \"kubernetes.io/projected/d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5-kube-api-access-tlcsc\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds\" (UID: \"d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds"
Jul 17 18:43:26.059747 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:26.059709    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds"
Jul 17 18:43:26.194557 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:26.194534    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds"]
Jul 17 18:43:26.195859 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:43:26.195831    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83d8a2a_28b5_4efd_b344_dbab5ddbb9b5.slice/crio-a2a7654b56bec09641614097a45137e9b82b3f26516aa41c383cc7da80a3a69f WatchSource:0}: Error finding container a2a7654b56bec09641614097a45137e9b82b3f26516aa41c383cc7da80a3a69f: Status 404 returned error can't find the container with id a2a7654b56bec09641614097a45137e9b82b3f26516aa41c383cc7da80a3a69f
Jul 17 18:43:26.624788 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:26.624757    2559 generic.go:356] "Generic (PLEG): container finished" podID="6624d4e9-b35a-4073-bda3-4dbf001ba7e0" containerID="b12a087076e7f0d0dd139f78e0a9650fb9ca43b85b50a6752978cff2e8c2440c" exitCode=0
Jul 17 18:43:26.625214 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:26.624839    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64" event={"ID":"6624d4e9-b35a-4073-bda3-4dbf001ba7e0","Type":"ContainerDied","Data":"b12a087076e7f0d0dd139f78e0a9650fb9ca43b85b50a6752978cff2e8c2440c"}
Jul 17 18:43:26.626534 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:26.626511    2559 generic.go:356] "Generic (PLEG): container finished" podID="d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5" containerID="2c8f3c38eff9bd1419bbb1b166b2bcbc32fa66943f1c4dbdb403a3fa03b875b2" exitCode=0
Jul 17 18:43:26.626696 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:26.626584    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds" event={"ID":"d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5","Type":"ContainerDied","Data":"2c8f3c38eff9bd1419bbb1b166b2bcbc32fa66943f1c4dbdb403a3fa03b875b2"}
Jul 17 18:43:26.626696 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:26.626634    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds" event={"ID":"d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5","Type":"ContainerStarted","Data":"a2a7654b56bec09641614097a45137e9b82b3f26516aa41c383cc7da80a3a69f"}
Jul 17 18:43:26.628152 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:26.628132    2559 generic.go:356] "Generic (PLEG): container finished" podID="1dc407dc-206b-44f2-9934-19d747560cb4" containerID="fd0269d3fa26a9162917d5f409a2cd3c8815f11577abeb19c6a4d776a7ffa94f" exitCode=0
Jul 17 18:43:26.628238 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:26.628171    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc" event={"ID":"1dc407dc-206b-44f2-9934-19d747560cb4","Type":"ContainerDied","Data":"fd0269d3fa26a9162917d5f409a2cd3c8815f11577abeb19c6a4d776a7ffa94f"}
Jul 17 18:43:26.628238 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:26.628200    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc" event={"ID":"1dc407dc-206b-44f2-9934-19d747560cb4","Type":"ContainerStarted","Data":"9a1c312163b739b507bfe1c69bd1112f0a8935749373a16df29dbdde1c1e21d9"}
Jul 17 18:43:27.634446 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:27.634411    2559 generic.go:356] "Generic (PLEG): container finished" podID="bfb82736-cf62-4f65-b8ae-d7f58162fce6" containerID="dae8ef0f9666f1518836a9a5292b135f3f9cc4fdbc8f5e74c513a42ef3263a8e" exitCode=0
Jul 17 18:43:27.634863 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:27.634487    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29" event={"ID":"bfb82736-cf62-4f65-b8ae-d7f58162fce6","Type":"ContainerDied","Data":"dae8ef0f9666f1518836a9a5292b135f3f9cc4fdbc8f5e74c513a42ef3263a8e"}
Jul 17 18:43:27.636727 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:27.636702    2559 generic.go:356] "Generic (PLEG): container finished" podID="6624d4e9-b35a-4073-bda3-4dbf001ba7e0" containerID="ba32bf8f937a0657fd568b2f598d0005bd20dd706c3978f8936dec3e2f0015a5" exitCode=0
Jul 17 18:43:27.636830 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:27.636745    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64" event={"ID":"6624d4e9-b35a-4073-bda3-4dbf001ba7e0","Type":"ContainerDied","Data":"ba32bf8f937a0657fd568b2f598d0005bd20dd706c3978f8936dec3e2f0015a5"}
Jul 17 18:43:27.638495 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:27.638475    2559 generic.go:356] "Generic (PLEG): container finished" podID="d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5" containerID="ba523fd7c89ae70cb6bb37fc9290148dde2068dcc58d4741e37368454460078d" exitCode=0
Jul 17 18:43:27.638584 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:27.638558    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds" event={"ID":"d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5","Type":"ContainerDied","Data":"ba523fd7c89ae70cb6bb37fc9290148dde2068dcc58d4741e37368454460078d"}
Jul 17 18:43:27.645383 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:27.642778    2559 generic.go:356] "Generic (PLEG): container finished" podID="1dc407dc-206b-44f2-9934-19d747560cb4" containerID="e9ace23482d6ccb0fc5eec89ec7f522f38c4d721f39565302cc31d342ed93255" exitCode=0
Jul 17 18:43:27.645383 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:27.642894    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc" event={"ID":"1dc407dc-206b-44f2-9934-19d747560cb4","Type":"ContainerDied","Data":"e9ace23482d6ccb0fc5eec89ec7f522f38c4d721f39565302cc31d342ed93255"}
Jul 17 18:43:28.650088 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:28.650055    2559 generic.go:356] "Generic (PLEG): container finished" podID="bfb82736-cf62-4f65-b8ae-d7f58162fce6" containerID="848e2f6d75a4946f8291eb87f1242acb45cb9e0116ae766bd5bb055cfc848d14" exitCode=0
Jul 17 18:43:28.650531 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:28.650130    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29" event={"ID":"bfb82736-cf62-4f65-b8ae-d7f58162fce6","Type":"ContainerDied","Data":"848e2f6d75a4946f8291eb87f1242acb45cb9e0116ae766bd5bb055cfc848d14"}
Jul 17 18:43:28.652081 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:28.652059    2559 generic.go:356] "Generic (PLEG): container finished" podID="d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5" containerID="702b571997aa3b7717a4d086a521cbd5c66a515a88e3ce4d89585fe61a2b7439" exitCode=0
Jul 17 18:43:28.652196 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:28.652148    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds" event={"ID":"d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5","Type":"ContainerDied","Data":"702b571997aa3b7717a4d086a521cbd5c66a515a88e3ce4d89585fe61a2b7439"}
Jul 17 18:43:28.654159 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:28.654138    2559 generic.go:356] "Generic (PLEG): container finished" podID="1dc407dc-206b-44f2-9934-19d747560cb4" containerID="f4268ef5f31112048551099bfc0d387260259b3e5aff1ba9b73a82c533680362" exitCode=0
Jul 17 18:43:28.654285 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:28.654198    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc" event={"ID":"1dc407dc-206b-44f2-9934-19d747560cb4","Type":"ContainerDied","Data":"f4268ef5f31112048551099bfc0d387260259b3e5aff1ba9b73a82c533680362"}
Jul 17 18:43:28.785222 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:28.785197    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64"
Jul 17 18:43:28.904035 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:28.903935    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6624d4e9-b35a-4073-bda3-4dbf001ba7e0-util\") pod \"6624d4e9-b35a-4073-bda3-4dbf001ba7e0\" (UID: \"6624d4e9-b35a-4073-bda3-4dbf001ba7e0\") "
Jul 17 18:43:28.904171 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:28.904041    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6624d4e9-b35a-4073-bda3-4dbf001ba7e0-bundle\") pod \"6624d4e9-b35a-4073-bda3-4dbf001ba7e0\" (UID: \"6624d4e9-b35a-4073-bda3-4dbf001ba7e0\") "
Jul 17 18:43:28.904171 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:28.904080    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwsk5\" (UniqueName: \"kubernetes.io/projected/6624d4e9-b35a-4073-bda3-4dbf001ba7e0-kube-api-access-dwsk5\") pod \"6624d4e9-b35a-4073-bda3-4dbf001ba7e0\" (UID: \"6624d4e9-b35a-4073-bda3-4dbf001ba7e0\") "
Jul 17 18:43:28.904641 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:28.904614    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6624d4e9-b35a-4073-bda3-4dbf001ba7e0-bundle" (OuterVolumeSpecName: "bundle") pod "6624d4e9-b35a-4073-bda3-4dbf001ba7e0" (UID: "6624d4e9-b35a-4073-bda3-4dbf001ba7e0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 18:43:28.906506 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:28.906482    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6624d4e9-b35a-4073-bda3-4dbf001ba7e0-kube-api-access-dwsk5" (OuterVolumeSpecName: "kube-api-access-dwsk5") pod "6624d4e9-b35a-4073-bda3-4dbf001ba7e0" (UID: "6624d4e9-b35a-4073-bda3-4dbf001ba7e0"). InnerVolumeSpecName "kube-api-access-dwsk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:43:28.909544 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:28.909509    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6624d4e9-b35a-4073-bda3-4dbf001ba7e0-util" (OuterVolumeSpecName: "util") pod "6624d4e9-b35a-4073-bda3-4dbf001ba7e0" (UID: "6624d4e9-b35a-4073-bda3-4dbf001ba7e0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 18:43:29.005136 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.005093    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6624d4e9-b35a-4073-bda3-4dbf001ba7e0-bundle\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:43:29.005136 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.005125    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwsk5\" (UniqueName: \"kubernetes.io/projected/6624d4e9-b35a-4073-bda3-4dbf001ba7e0-kube-api-access-dwsk5\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:43:29.005136 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.005143    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6624d4e9-b35a-4073-bda3-4dbf001ba7e0-util\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:43:29.661074 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.661044    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64"
Jul 17 18:43:29.661074 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.661068    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64" event={"ID":"6624d4e9-b35a-4073-bda3-4dbf001ba7e0","Type":"ContainerDied","Data":"5ddfca921237945ccfb2b3d1584aad97bfce59256d2824678ff2da1af4ca625e"}
Jul 17 18:43:29.661541 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.661116    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ddfca921237945ccfb2b3d1584aad97bfce59256d2824678ff2da1af4ca625e"
Jul 17 18:43:29.798485 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.798455    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds"
Jul 17 18:43:29.852630 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.852602    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29"
Jul 17 18:43:29.855650 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.855629    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc"
Jul 17 18:43:29.912228 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.912152    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5-util\") pod \"d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5\" (UID: \"d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5\") "
Jul 17 18:43:29.912228 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.912188    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-md4tt\" (UniqueName: \"kubernetes.io/projected/1dc407dc-206b-44f2-9934-19d747560cb4-kube-api-access-md4tt\") pod \"1dc407dc-206b-44f2-9934-19d747560cb4\" (UID: \"1dc407dc-206b-44f2-9934-19d747560cb4\") "
Jul 17 18:43:29.912228 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.912225    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfb82736-cf62-4f65-b8ae-d7f58162fce6-util\") pod \"bfb82736-cf62-4f65-b8ae-d7f58162fce6\" (UID: \"bfb82736-cf62-4f65-b8ae-d7f58162fce6\") "
Jul 17 18:43:29.912533 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.912249    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1dc407dc-206b-44f2-9934-19d747560cb4-util\") pod \"1dc407dc-206b-44f2-9934-19d747560cb4\" (UID: \"1dc407dc-206b-44f2-9934-19d747560cb4\") "
Jul 17 18:43:29.912533 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.912275    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlxjw\" (UniqueName: \"kubernetes.io/projected/bfb82736-cf62-4f65-b8ae-d7f58162fce6-kube-api-access-mlxjw\") pod \"bfb82736-cf62-4f65-b8ae-d7f58162fce6\" (UID: \"bfb82736-cf62-4f65-b8ae-d7f58162fce6\") "
Jul 17 18:43:29.912533 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.912334    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1dc407dc-206b-44f2-9934-19d747560cb4-bundle\") pod \"1dc407dc-206b-44f2-9934-19d747560cb4\" (UID: \"1dc407dc-206b-44f2-9934-19d747560cb4\") "
Jul 17 18:43:29.912533 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.912369    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5-bundle\") pod \"d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5\" (UID: \"d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5\") "
Jul 17 18:43:29.912533 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.912397    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfb82736-cf62-4f65-b8ae-d7f58162fce6-bundle\") pod \"bfb82736-cf62-4f65-b8ae-d7f58162fce6\" (UID: \"bfb82736-cf62-4f65-b8ae-d7f58162fce6\") "
Jul 17 18:43:29.912533 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.912442    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlcsc\" (UniqueName: \"kubernetes.io/projected/d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5-kube-api-access-tlcsc\") pod \"d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5\" (UID: \"d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5\") "
Jul 17 18:43:29.914114 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.913532    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc407dc-206b-44f2-9934-19d747560cb4-bundle" (OuterVolumeSpecName: "bundle") pod "1dc407dc-206b-44f2-9934-19d747560cb4" (UID: "1dc407dc-206b-44f2-9934-19d747560cb4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 18:43:29.914114 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.913677    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5-bundle" (OuterVolumeSpecName: "bundle") pod "d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5" (UID: "d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 18:43:29.914114 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.914081    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb82736-cf62-4f65-b8ae-d7f58162fce6-bundle" (OuterVolumeSpecName: "bundle") pod "bfb82736-cf62-4f65-b8ae-d7f58162fce6" (UID: "bfb82736-cf62-4f65-b8ae-d7f58162fce6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 18:43:29.915495 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.915470    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5-kube-api-access-tlcsc" (OuterVolumeSpecName: "kube-api-access-tlcsc") pod "d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5" (UID: "d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5"). InnerVolumeSpecName "kube-api-access-tlcsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:43:29.915893 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.915858    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb82736-cf62-4f65-b8ae-d7f58162fce6-kube-api-access-mlxjw" (OuterVolumeSpecName: "kube-api-access-mlxjw") pod "bfb82736-cf62-4f65-b8ae-d7f58162fce6" (UID: "bfb82736-cf62-4f65-b8ae-d7f58162fce6"). InnerVolumeSpecName "kube-api-access-mlxjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:43:29.915967 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.915900    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc407dc-206b-44f2-9934-19d747560cb4-kube-api-access-md4tt" (OuterVolumeSpecName: "kube-api-access-md4tt") pod "1dc407dc-206b-44f2-9934-19d747560cb4" (UID: "1dc407dc-206b-44f2-9934-19d747560cb4"). InnerVolumeSpecName "kube-api-access-md4tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:43:29.918633 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.918616    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb82736-cf62-4f65-b8ae-d7f58162fce6-util" (OuterVolumeSpecName: "util") pod "bfb82736-cf62-4f65-b8ae-d7f58162fce6" (UID: "bfb82736-cf62-4f65-b8ae-d7f58162fce6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 18:43:29.920680 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.920654    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5-util" (OuterVolumeSpecName: "util") pod "d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5" (UID: "d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 18:43:29.921151 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:29.921123    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc407dc-206b-44f2-9934-19d747560cb4-util" (OuterVolumeSpecName: "util") pod "1dc407dc-206b-44f2-9934-19d747560cb4" (UID: "1dc407dc-206b-44f2-9934-19d747560cb4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 18:43:30.013625 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:30.013588    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfb82736-cf62-4f65-b8ae-d7f58162fce6-util\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:43:30.013625 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:30.013617    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1dc407dc-206b-44f2-9934-19d747560cb4-util\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:43:30.013625 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:30.013626    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlxjw\" (UniqueName: \"kubernetes.io/projected/bfb82736-cf62-4f65-b8ae-d7f58162fce6-kube-api-access-mlxjw\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:43:30.013625 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:30.013636    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1dc407dc-206b-44f2-9934-19d747560cb4-bundle\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:43:30.013891 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:30.013645    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5-bundle\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:43:30.013891 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:30.013652    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfb82736-cf62-4f65-b8ae-d7f58162fce6-bundle\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:43:30.013891 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:30.013659    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlcsc\" (UniqueName: \"kubernetes.io/projected/d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5-kube-api-access-tlcsc\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:43:30.013891 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:30.013667    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5-util\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:43:30.013891 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:30.013675    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md4tt\" (UniqueName: \"kubernetes.io/projected/1dc407dc-206b-44f2-9934-19d747560cb4-kube-api-access-md4tt\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:43:30.668270 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:30.668234    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29" event={"ID":"bfb82736-cf62-4f65-b8ae-d7f58162fce6","Type":"ContainerDied","Data":"8c4c7ee5bf26a12cc76cabf64f4edc41fe4df3822fd95c11ca70746c568ecb0b"}
Jul 17 18:43:30.668706 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:30.668244    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29"
Jul 17 18:43:30.668706 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:30.668343    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c4c7ee5bf26a12cc76cabf64f4edc41fe4df3822fd95c11ca70746c568ecb0b"
Jul 17 18:43:30.670588 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:30.670547    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds"
Jul 17 18:43:30.670707 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:30.670570    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds" event={"ID":"d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5","Type":"ContainerDied","Data":"a2a7654b56bec09641614097a45137e9b82b3f26516aa41c383cc7da80a3a69f"}
Jul 17 18:43:30.670707 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:30.670625    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2a7654b56bec09641614097a45137e9b82b3f26516aa41c383cc7da80a3a69f"
Jul 17 18:43:30.672589 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:30.672532    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc" event={"ID":"1dc407dc-206b-44f2-9934-19d747560cb4","Type":"ContainerDied","Data":"9a1c312163b739b507bfe1c69bd1112f0a8935749373a16df29dbdde1c1e21d9"}
Jul 17 18:43:30.672589 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:30.672572    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a1c312163b739b507bfe1c69bd1112f0a8935749373a16df29dbdde1c1e21d9"
Jul 17 18:43:30.672589 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:30.672575    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc"
Jul 17 18:43:56.162989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.162951    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pqdnf"]
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163298    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc407dc-206b-44f2-9934-19d747560cb4" containerName="pull"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163326    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc407dc-206b-44f2-9934-19d747560cb4" containerName="pull"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163348    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6624d4e9-b35a-4073-bda3-4dbf001ba7e0" containerName="util"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163354    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6624d4e9-b35a-4073-bda3-4dbf001ba7e0" containerName="util"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163361    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb82736-cf62-4f65-b8ae-d7f58162fce6" containerName="pull"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163366    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb82736-cf62-4f65-b8ae-d7f58162fce6" containerName="pull"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163371    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc407dc-206b-44f2-9934-19d747560cb4" containerName="util"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163376    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc407dc-206b-44f2-9934-19d747560cb4" containerName="util"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163387    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6624d4e9-b35a-4073-bda3-4dbf001ba7e0" containerName="extract"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163393    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6624d4e9-b35a-4073-bda3-4dbf001ba7e0" containerName="extract"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163402    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc407dc-206b-44f2-9934-19d747560cb4" containerName="extract"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163407    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc407dc-206b-44f2-9934-19d747560cb4" containerName="extract"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163414    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5" containerName="pull"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163419    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5" containerName="pull"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163427    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb82736-cf62-4f65-b8ae-d7f58162fce6" containerName="util"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163431    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb82736-cf62-4f65-b8ae-d7f58162fce6" containerName="util"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163438    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5" containerName="extract"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163443    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5" containerName="extract"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163449    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5" containerName="util"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163454    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5" containerName="util"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163460    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6624d4e9-b35a-4073-bda3-4dbf001ba7e0" containerName="pull"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163464    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6624d4e9-b35a-4073-bda3-4dbf001ba7e0" containerName="pull"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163469    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb82736-cf62-4f65-b8ae-d7f58162fce6" containerName="extract"
Jul 17 18:43:56.163497 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163474    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb82736-cf62-4f65-b8ae-d7f58162fce6" containerName="extract"
Jul 17 18:43:56.164203 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163527    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5" containerName="extract"
Jul 17 18:43:56.164203 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163538    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc407dc-206b-44f2-9934-19d747560cb4" containerName="extract"
Jul 17 18:43:56.164203 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163546    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6624d4e9-b35a-4073-bda3-4dbf001ba7e0" containerName="extract"
Jul 17 18:43:56.164203 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.163552    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb82736-cf62-4f65-b8ae-d7f58162fce6" containerName="extract"
Jul 17 18:43:56.166580 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.166564    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pqdnf"
Jul 17 18:43:56.169233 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.169203    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 18:43:56.169233 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.169207    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 17 18:43:56.169459 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.169274    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 17 18:43:56.170142 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.170122    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tzbmh\""
Jul 17 18:43:56.170236 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.170144    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 18:43:56.178689 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.178667    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pqdnf"]
Jul 17 18:43:56.246765 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.246723    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9138212c-21f1-414e-a2c3-d726e2d8b609-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pqdnf\" (UID: \"9138212c-21f1-414e-a2c3-d726e2d8b609\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pqdnf"
Jul 17 18:43:56.246954 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.246766    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9138212c-21f1-414e-a2c3-d726e2d8b609-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pqdnf\" (UID: \"9138212c-21f1-414e-a2c3-d726e2d8b609\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pqdnf"
Jul 17 18:43:56.246954 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.246813    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d856q\" (UniqueName: \"kubernetes.io/projected/9138212c-21f1-414e-a2c3-d726e2d8b609-kube-api-access-d856q\") pod \"kuadrant-console-plugin-66b5774985-pqdnf\" (UID: \"9138212c-21f1-414e-a2c3-d726e2d8b609\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pqdnf"
Jul 17 18:43:56.348333 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.348270    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9138212c-21f1-414e-a2c3-d726e2d8b609-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pqdnf\" (UID: \"9138212c-21f1-414e-a2c3-d726e2d8b609\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pqdnf"
Jul 17 18:43:56.348333 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.348333    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9138212c-21f1-414e-a2c3-d726e2d8b609-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pqdnf\" (UID: \"9138212c-21f1-414e-a2c3-d726e2d8b609\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pqdnf"
Jul 17 18:43:56.348550 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.348364    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d856q\" (UniqueName: \"kubernetes.io/projected/9138212c-21f1-414e-a2c3-d726e2d8b609-kube-api-access-d856q\") pod \"kuadrant-console-plugin-66b5774985-pqdnf\" (UID: \"9138212c-21f1-414e-a2c3-d726e2d8b609\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pqdnf"
Jul 17 18:43:56.349048 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.349027    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9138212c-21f1-414e-a2c3-d726e2d8b609-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pqdnf\" (UID: \"9138212c-21f1-414e-a2c3-d726e2d8b609\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pqdnf"
Jul 17 18:43:56.350900 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.350865    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9138212c-21f1-414e-a2c3-d726e2d8b609-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pqdnf\" (UID: \"9138212c-21f1-414e-a2c3-d726e2d8b609\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pqdnf"
Jul 17 18:43:56.358454 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.358433    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d856q\" (UniqueName: \"kubernetes.io/projected/9138212c-21f1-414e-a2c3-d726e2d8b609-kube-api-access-d856q\") pod \"kuadrant-console-plugin-66b5774985-pqdnf\" (UID: \"9138212c-21f1-414e-a2c3-d726e2d8b609\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pqdnf"
Jul 17 18:43:56.476564 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.476482    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pqdnf"
Jul 17 18:43:56.627559 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.627528    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pqdnf"]
Jul 17 18:43:56.629569 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:43:56.629542    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9138212c_21f1_414e_a2c3_d726e2d8b609.slice/crio-ecfc340be1c4106f0e51d0881bf701e85bad74253d8b41c69e8671fc0e057a0e WatchSource:0}: Error finding container ecfc340be1c4106f0e51d0881bf701e85bad74253d8b41c69e8671fc0e057a0e: Status 404 returned error can't find the container with id ecfc340be1c4106f0e51d0881bf701e85bad74253d8b41c69e8671fc0e057a0e
Jul 17 18:43:56.811057 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:43:56.811021    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pqdnf" event={"ID":"9138212c-21f1-414e-a2c3-d726e2d8b609","Type":"ContainerStarted","Data":"ecfc340be1c4106f0e51d0881bf701e85bad74253d8b41c69e8671fc0e057a0e"}
Jul 17 18:44:18.945287 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:18.945195    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pqdnf" event={"ID":"9138212c-21f1-414e-a2c3-d726e2d8b609","Type":"ContainerStarted","Data":"718412fd540a9f1e4f652a75a373e2b0b7871c839b7ce16e82c8576c42f928a6"}
Jul 17 18:44:18.968251 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:18.968204    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pqdnf" podStartSLOduration=0.924787311 podStartE2EDuration="22.96818897s" totalImagesPullingTime="22.043401659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:43:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:18.967993399 +0000 UTC m=+669.409502091" watchObservedRunningTime="2026-07-17 18:44:18.96818897 +0000 UTC m=+669.409697659"
Jul 17 18:44:39.974177 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:39.974139    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75755b7754-qmvld"]
Jul 17 18:44:39.997516 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:39.997486    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75755b7754-qmvld"]
Jul 17 18:44:39.997666 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:39.997613    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.000387 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.000362    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 17 18:44:40.001260 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.001239    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nt4s4\""
Jul 17 18:44:40.001419 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.001324    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 17 18:44:40.001501 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.001490    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 17 18:44:40.001560 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.001542    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 17 18:44:40.002567 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.002541    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 17 18:44:40.005579 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.005559    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 17 18:44:40.160772 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.160739    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8fb8bab-54a3-4811-869c-122ff53c8e5f-trusted-ca-bundle\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.160935 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.160795    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8fb8bab-54a3-4811-869c-122ff53c8e5f-console-serving-cert\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.160935 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.160814    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv7x7\" (UniqueName: \"kubernetes.io/projected/a8fb8bab-54a3-4811-869c-122ff53c8e5f-kube-api-access-gv7x7\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.160935 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.160842    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8fb8bab-54a3-4811-869c-122ff53c8e5f-oauth-serving-cert\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.160935 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.160883    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8fb8bab-54a3-4811-869c-122ff53c8e5f-service-ca\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.160935 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.160909    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8fb8bab-54a3-4811-869c-122ff53c8e5f-console-oauth-config\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.160935 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.160935    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8fb8bab-54a3-4811-869c-122ff53c8e5f-console-config\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.261490 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.261412    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8fb8bab-54a3-4811-869c-122ff53c8e5f-console-serving-cert\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.261490 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.261444    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv7x7\" (UniqueName: \"kubernetes.io/projected/a8fb8bab-54a3-4811-869c-122ff53c8e5f-kube-api-access-gv7x7\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.261490 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.261481    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8fb8bab-54a3-4811-869c-122ff53c8e5f-oauth-serving-cert\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.261785 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.261517    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8fb8bab-54a3-4811-869c-122ff53c8e5f-service-ca\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.261785 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.261547    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8fb8bab-54a3-4811-869c-122ff53c8e5f-console-oauth-config\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.261785 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.261585    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8fb8bab-54a3-4811-869c-122ff53c8e5f-console-config\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.261785 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.261625    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8fb8bab-54a3-4811-869c-122ff53c8e5f-trusted-ca-bundle\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.262364 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.262300    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8fb8bab-54a3-4811-869c-122ff53c8e5f-oauth-serving-cert\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.262483 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.262421    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8fb8bab-54a3-4811-869c-122ff53c8e5f-service-ca\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.262483 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.262426    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8fb8bab-54a3-4811-869c-122ff53c8e5f-console-config\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.262777 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.262756    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8fb8bab-54a3-4811-869c-122ff53c8e5f-trusted-ca-bundle\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.264133 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.264112    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8fb8bab-54a3-4811-869c-122ff53c8e5f-console-oauth-config\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.264286 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.264268    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8fb8bab-54a3-4811-869c-122ff53c8e5f-console-serving-cert\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.272664 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.272640    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv7x7\" (UniqueName: \"kubernetes.io/projected/a8fb8bab-54a3-4811-869c-122ff53c8e5f-kube-api-access-gv7x7\") pod \"console-75755b7754-qmvld\" (UID: \"a8fb8bab-54a3-4811-869c-122ff53c8e5f\") " pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.308648 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.308618    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:40.502293 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:40.502272    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75755b7754-qmvld"]
Jul 17 18:44:40.505213 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:44:40.505186    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8fb8bab_54a3_4811_869c_122ff53c8e5f.slice/crio-5137e876c8bb08ba1877df211199337521ba88f2db57a02a406663ab6bb043e8 WatchSource:0}: Error finding container 5137e876c8bb08ba1877df211199337521ba88f2db57a02a406663ab6bb043e8: Status 404 returned error can't find the container with id 5137e876c8bb08ba1877df211199337521ba88f2db57a02a406663ab6bb043e8
Jul 17 18:44:41.056834 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:41.056791    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75755b7754-qmvld" event={"ID":"a8fb8bab-54a3-4811-869c-122ff53c8e5f","Type":"ContainerStarted","Data":"9851789ad60509e20d615bfbd425e6e2e93cca3108ccb3c5ec33e983ca46b17d"}
Jul 17 18:44:41.057208 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:41.056887    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75755b7754-qmvld" event={"ID":"a8fb8bab-54a3-4811-869c-122ff53c8e5f","Type":"ContainerStarted","Data":"5137e876c8bb08ba1877df211199337521ba88f2db57a02a406663ab6bb043e8"}
Jul 17 18:44:41.077018 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:41.076962    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75755b7754-qmvld" podStartSLOduration=2.076946959 podStartE2EDuration="2.076946959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:41.074346847 +0000 UTC m=+691.515855535" watchObservedRunningTime="2026-07-17 18:44:41.076946959 +0000 UTC m=+691.518455647"
Jul 17 18:44:50.309507 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:50.309462    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:50.310097 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:50.309545    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:50.314110 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:50.314089    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:50.860396 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:50.860360    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-v8kkv"]
Jul 17 18:44:50.890791 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:50.890754    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-v8kkv"]
Jul 17 18:44:50.890956 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:50.890851    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-v8kkv"
Jul 17 18:44:50.893374 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:50.893348    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 17 18:44:50.949632 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:50.949597    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-v8kkv"]
Jul 17 18:44:50.956489 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:50.956463    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e5597bfc-6629-45b8-afc9-7d73c7726806-config-file\") pod \"limitador-limitador-64c94fd64f-v8kkv\" (UID: \"e5597bfc-6629-45b8-afc9-7d73c7726806\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-v8kkv"
Jul 17 18:44:50.956707 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:50.956573    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p25dh\" (UniqueName: \"kubernetes.io/projected/e5597bfc-6629-45b8-afc9-7d73c7726806-kube-api-access-p25dh\") pod \"limitador-limitador-64c94fd64f-v8kkv\" (UID: \"e5597bfc-6629-45b8-afc9-7d73c7726806\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-v8kkv"
Jul 17 18:44:51.057474 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:51.057443    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e5597bfc-6629-45b8-afc9-7d73c7726806-config-file\") pod \"limitador-limitador-64c94fd64f-v8kkv\" (UID: \"e5597bfc-6629-45b8-afc9-7d73c7726806\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-v8kkv"
Jul 17 18:44:51.057653 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:51.057520    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p25dh\" (UniqueName: \"kubernetes.io/projected/e5597bfc-6629-45b8-afc9-7d73c7726806-kube-api-access-p25dh\") pod \"limitador-limitador-64c94fd64f-v8kkv\" (UID: \"e5597bfc-6629-45b8-afc9-7d73c7726806\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-v8kkv"
Jul 17 18:44:51.058092 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:51.058070    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e5597bfc-6629-45b8-afc9-7d73c7726806-config-file\") pod \"limitador-limitador-64c94fd64f-v8kkv\" (UID: \"e5597bfc-6629-45b8-afc9-7d73c7726806\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-v8kkv"
Jul 17 18:44:51.066563 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:51.066535    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p25dh\" (UniqueName: \"kubernetes.io/projected/e5597bfc-6629-45b8-afc9-7d73c7726806-kube-api-access-p25dh\") pod \"limitador-limitador-64c94fd64f-v8kkv\" (UID: \"e5597bfc-6629-45b8-afc9-7d73c7726806\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-v8kkv"
Jul 17 18:44:51.113496 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:51.113422    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75755b7754-qmvld"
Jul 17 18:44:51.203657 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:51.203630    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-v8kkv"
Jul 17 18:44:51.350455 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:51.350430    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-v8kkv"]
Jul 17 18:44:51.352301 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:44:51.352271    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5597bfc_6629_45b8_afc9_7d73c7726806.slice/crio-136d908ff005a5c5b665a832fd544590bae7a3969888b9cbd180c6ade4731ba5 WatchSource:0}: Error finding container 136d908ff005a5c5b665a832fd544590bae7a3969888b9cbd180c6ade4731ba5: Status 404 returned error can't find the container with id 136d908ff005a5c5b665a832fd544590bae7a3969888b9cbd180c6ade4731ba5
Jul 17 18:44:52.116375 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:52.116337    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-v8kkv" event={"ID":"e5597bfc-6629-45b8-afc9-7d73c7726806","Type":"ContainerStarted","Data":"136d908ff005a5c5b665a832fd544590bae7a3969888b9cbd180c6ade4731ba5"}
Jul 17 18:44:55.137636 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:55.137604    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-v8kkv" event={"ID":"e5597bfc-6629-45b8-afc9-7d73c7726806","Type":"ContainerStarted","Data":"45ff4f47d67e926168cbb0f4e08b7e57d10a5851ee53ebec99328182846f175c"}
Jul 17 18:44:55.138071 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:55.137686    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-v8kkv"
Jul 17 18:44:55.158746 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:44:55.158700    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-v8kkv" podStartSLOduration=2.358324897 podStartE2EDuration="5.158686273s" totalImagesPullingTime="2.800361376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:55.156322164 +0000 UTC m=+705.597830843" watchObservedRunningTime="2026-07-17 18:44:55.158686273 +0000 UTC m=+705.600194962"
Jul 17 18:45:06.144155 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:06.144083    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-v8kkv"
Jul 17 18:45:06.161527 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:06.161498    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-v8kkv"]
Jul 17 18:45:06.191368 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:06.191331    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-v8kkv" podUID="e5597bfc-6629-45b8-afc9-7d73c7726806" containerName="limitador" containerID="cri-o://45ff4f47d67e926168cbb0f4e08b7e57d10a5851ee53ebec99328182846f175c" gracePeriod=30
Jul 17 18:45:06.739345 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:06.739298    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-v8kkv"
Jul 17 18:45:06.794616 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:06.794532    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p25dh\" (UniqueName: \"kubernetes.io/projected/e5597bfc-6629-45b8-afc9-7d73c7726806-kube-api-access-p25dh\") pod \"e5597bfc-6629-45b8-afc9-7d73c7726806\" (UID: \"e5597bfc-6629-45b8-afc9-7d73c7726806\") "
Jul 17 18:45:06.794772 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:06.794648    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e5597bfc-6629-45b8-afc9-7d73c7726806-config-file\") pod \"e5597bfc-6629-45b8-afc9-7d73c7726806\" (UID: \"e5597bfc-6629-45b8-afc9-7d73c7726806\") "
Jul 17 18:45:06.795060 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:06.795031    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5597bfc-6629-45b8-afc9-7d73c7726806-config-file" (OuterVolumeSpecName: "config-file") pod "e5597bfc-6629-45b8-afc9-7d73c7726806" (UID: "e5597bfc-6629-45b8-afc9-7d73c7726806"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:45:06.796676 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:06.796649    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5597bfc-6629-45b8-afc9-7d73c7726806-kube-api-access-p25dh" (OuterVolumeSpecName: "kube-api-access-p25dh") pod "e5597bfc-6629-45b8-afc9-7d73c7726806" (UID: "e5597bfc-6629-45b8-afc9-7d73c7726806"). InnerVolumeSpecName "kube-api-access-p25dh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:45:06.895934 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:06.895897    2559 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e5597bfc-6629-45b8-afc9-7d73c7726806-config-file\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:45:06.895934 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:06.895927    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p25dh\" (UniqueName: \"kubernetes.io/projected/e5597bfc-6629-45b8-afc9-7d73c7726806-kube-api-access-p25dh\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:45:07.197819 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:07.197786    2559 generic.go:356] "Generic (PLEG): container finished" podID="e5597bfc-6629-45b8-afc9-7d73c7726806" containerID="45ff4f47d67e926168cbb0f4e08b7e57d10a5851ee53ebec99328182846f175c" exitCode=0
Jul 17 18:45:07.198271 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:07.197830    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-v8kkv" event={"ID":"e5597bfc-6629-45b8-afc9-7d73c7726806","Type":"ContainerDied","Data":"45ff4f47d67e926168cbb0f4e08b7e57d10a5851ee53ebec99328182846f175c"}
Jul 17 18:45:07.198271 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:07.197843    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-v8kkv"
Jul 17 18:45:07.198271 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:07.197890    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-v8kkv" event={"ID":"e5597bfc-6629-45b8-afc9-7d73c7726806","Type":"ContainerDied","Data":"136d908ff005a5c5b665a832fd544590bae7a3969888b9cbd180c6ade4731ba5"}
Jul 17 18:45:07.198271 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:07.197914    2559 scope.go:117] "RemoveContainer" containerID="45ff4f47d67e926168cbb0f4e08b7e57d10a5851ee53ebec99328182846f175c"
Jul 17 18:45:07.207609 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:07.207584    2559 scope.go:117] "RemoveContainer" containerID="45ff4f47d67e926168cbb0f4e08b7e57d10a5851ee53ebec99328182846f175c"
Jul 17 18:45:07.207899 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:45:07.207879    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ff4f47d67e926168cbb0f4e08b7e57d10a5851ee53ebec99328182846f175c\": container with ID starting with 45ff4f47d67e926168cbb0f4e08b7e57d10a5851ee53ebec99328182846f175c not found: ID does not exist" containerID="45ff4f47d67e926168cbb0f4e08b7e57d10a5851ee53ebec99328182846f175c"
Jul 17 18:45:07.207966 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:07.207909    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ff4f47d67e926168cbb0f4e08b7e57d10a5851ee53ebec99328182846f175c"} err="failed to get container status \"45ff4f47d67e926168cbb0f4e08b7e57d10a5851ee53ebec99328182846f175c\": rpc error: code = NotFound desc = could not find container \"45ff4f47d67e926168cbb0f4e08b7e57d10a5851ee53ebec99328182846f175c\": container with ID starting with 45ff4f47d67e926168cbb0f4e08b7e57d10a5851ee53ebec99328182846f175c not found: ID does not exist"
Jul 17 18:45:07.219759 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:07.219734    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-v8kkv"]
Jul 17 18:45:07.221768 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:07.221742    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-v8kkv"]
Jul 17 18:45:08.125476 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:08.125447    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5597bfc-6629-45b8-afc9-7d73c7726806" path="/var/lib/kubelet/pods/e5597bfc-6629-45b8-afc9-7d73c7726806/volumes"
Jul 17 18:45:09.460321 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:09.460286    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qcm76"]
Jul 17 18:45:09.460717 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:09.460636    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5597bfc-6629-45b8-afc9-7d73c7726806" containerName="limitador"
Jul 17 18:45:09.460717 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:09.460648    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5597bfc-6629-45b8-afc9-7d73c7726806" containerName="limitador"
Jul 17 18:45:09.460717 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:09.460717    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5597bfc-6629-45b8-afc9-7d73c7726806" containerName="limitador"
Jul 17 18:45:09.463665 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:09.463645    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcm76"
Jul 17 18:45:09.466689 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:09.466667    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 18:45:09.467680 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:09.467659    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-h7c4l\""
Jul 17 18:45:09.467802 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:09.467675    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 18:45:09.467802 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:09.467667    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 17 18:45:09.478196 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:09.478174    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qcm76"]
Jul 17 18:45:09.520403 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:09.520376    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwh9j\" (UniqueName: \"kubernetes.io/projected/7aba07c7-e794-46d0-887a-0171d5250c5f-kube-api-access-nwh9j\") pod \"postgres-69fb4594cf-qcm76\" (UID: \"7aba07c7-e794-46d0-887a-0171d5250c5f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcm76"
Jul 17 18:45:09.520566 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:09.520464    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7aba07c7-e794-46d0-887a-0171d5250c5f-data\") pod \"postgres-69fb4594cf-qcm76\" (UID: \"7aba07c7-e794-46d0-887a-0171d5250c5f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcm76"
Jul 17 18:45:09.621654 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:09.621604    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7aba07c7-e794-46d0-887a-0171d5250c5f-data\") pod \"postgres-69fb4594cf-qcm76\" (UID: \"7aba07c7-e794-46d0-887a-0171d5250c5f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcm76"
Jul 17 18:45:09.621825 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:09.621674    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwh9j\" (UniqueName: \"kubernetes.io/projected/7aba07c7-e794-46d0-887a-0171d5250c5f-kube-api-access-nwh9j\") pod \"postgres-69fb4594cf-qcm76\" (UID: \"7aba07c7-e794-46d0-887a-0171d5250c5f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcm76"
Jul 17 18:45:09.622065 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:09.622046    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7aba07c7-e794-46d0-887a-0171d5250c5f-data\") pod \"postgres-69fb4594cf-qcm76\" (UID: \"7aba07c7-e794-46d0-887a-0171d5250c5f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcm76"
Jul 17 18:45:09.631046 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:09.631023    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwh9j\" (UniqueName: \"kubernetes.io/projected/7aba07c7-e794-46d0-887a-0171d5250c5f-kube-api-access-nwh9j\") pod \"postgres-69fb4594cf-qcm76\" (UID: \"7aba07c7-e794-46d0-887a-0171d5250c5f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcm76"
Jul 17 18:45:09.774402 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:09.774295    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcm76"
Jul 17 18:45:09.912741 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:09.912715    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qcm76"]
Jul 17 18:45:09.914526 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:45:09.914496    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aba07c7_e794_46d0_887a_0171d5250c5f.slice/crio-613b24425db49a787655488758f7117fa5f5fb2cac1fd569cdf4c5755df7ac6e WatchSource:0}: Error finding container 613b24425db49a787655488758f7117fa5f5fb2cac1fd569cdf4c5755df7ac6e: Status 404 returned error can't find the container with id 613b24425db49a787655488758f7117fa5f5fb2cac1fd569cdf4c5755df7ac6e
Jul 17 18:45:10.216354 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:10.216321    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcm76" event={"ID":"7aba07c7-e794-46d0-887a-0171d5250c5f","Type":"ContainerStarted","Data":"613b24425db49a787655488758f7117fa5f5fb2cac1fd569cdf4c5755df7ac6e"}
Jul 17 18:45:14.978767 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:14.978745    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 17 18:45:15.250109 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:15.250021    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcm76" event={"ID":"7aba07c7-e794-46d0-887a-0171d5250c5f","Type":"ContainerStarted","Data":"2f117e3aa4acdeb0febba0c1d8a495d684b4ecb3ac0be8342a6bf54a5d12612d"}
Jul 17 18:45:15.250109 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:15.250089    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcm76"
Jul 17 18:45:15.267032 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:15.266988    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcm76" podStartSLOduration=1.207336443 podStartE2EDuration="6.266975854s" totalImagesPullingTime="5.059639411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:45:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:15.265603168 +0000 UTC m=+725.707111858" watchObservedRunningTime="2026-07-17 18:45:15.266975854 +0000 UTC m=+725.708484543"
Jul 17 18:45:21.287827 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:21.287796    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qcm76"
Jul 17 18:45:28.620695 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:28.620660    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c84488dd8-sb8dc"]
Jul 17 18:45:28.641964 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:28.641930    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c84488dd8-sb8dc"]
Jul 17 18:45:28.642119 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:28.642063    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c84488dd8-sb8dc"
Jul 17 18:45:28.645841 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:28.645813    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 17 18:45:28.645841 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:28.645823    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8hd7g\""
Jul 17 18:45:28.646112 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:28.645823    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 17 18:45:28.795460 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:28.795420    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdq8f\" (UniqueName: \"kubernetes.io/projected/90b5e339-e754-42ff-96e1-e9d04d5570d0-kube-api-access-wdq8f\") pod \"maas-controller-7c84488dd8-sb8dc\" (UID: \"90b5e339-e754-42ff-96e1-e9d04d5570d0\") " pod="opendatahub/maas-controller-7c84488dd8-sb8dc"
Jul 17 18:45:28.795659 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:28.795553    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90b5e339-e754-42ff-96e1-e9d04d5570d0-webhook-cert\") pod \"maas-controller-7c84488dd8-sb8dc\" (UID: \"90b5e339-e754-42ff-96e1-e9d04d5570d0\") " pod="opendatahub/maas-controller-7c84488dd8-sb8dc"
Jul 17 18:45:28.896239 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:28.896159    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdq8f\" (UniqueName: \"kubernetes.io/projected/90b5e339-e754-42ff-96e1-e9d04d5570d0-kube-api-access-wdq8f\") pod \"maas-controller-7c84488dd8-sb8dc\" (UID: \"90b5e339-e754-42ff-96e1-e9d04d5570d0\") " pod="opendatahub/maas-controller-7c84488dd8-sb8dc"
Jul 17 18:45:28.896405 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:28.896267    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90b5e339-e754-42ff-96e1-e9d04d5570d0-webhook-cert\") pod \"maas-controller-7c84488dd8-sb8dc\" (UID: \"90b5e339-e754-42ff-96e1-e9d04d5570d0\") " pod="opendatahub/maas-controller-7c84488dd8-sb8dc"
Jul 17 18:45:28.898900 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:28.898866    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90b5e339-e754-42ff-96e1-e9d04d5570d0-webhook-cert\") pod \"maas-controller-7c84488dd8-sb8dc\" (UID: \"90b5e339-e754-42ff-96e1-e9d04d5570d0\") " pod="opendatahub/maas-controller-7c84488dd8-sb8dc"
Jul 17 18:45:28.906344 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:28.906191    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdq8f\" (UniqueName: \"kubernetes.io/projected/90b5e339-e754-42ff-96e1-e9d04d5570d0-kube-api-access-wdq8f\") pod \"maas-controller-7c84488dd8-sb8dc\" (UID: \"90b5e339-e754-42ff-96e1-e9d04d5570d0\") " pod="opendatahub/maas-controller-7c84488dd8-sb8dc"
Jul 17 18:45:28.952921 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:28.952884    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c84488dd8-sb8dc"
Jul 17 18:45:29.087642 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:29.087613    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c84488dd8-sb8dc"]
Jul 17 18:45:29.090929 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:45:29.090903    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b5e339_e754_42ff_96e1_e9d04d5570d0.slice/crio-f99979719b31a3f096bb0b4a3f651d7e29a2de4d9fb4d37ec6baacf3071fc84d WatchSource:0}: Error finding container f99979719b31a3f096bb0b4a3f651d7e29a2de4d9fb4d37ec6baacf3071fc84d: Status 404 returned error can't find the container with id f99979719b31a3f096bb0b4a3f651d7e29a2de4d9fb4d37ec6baacf3071fc84d
Jul 17 18:45:29.330380 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:29.330345    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c84488dd8-sb8dc" event={"ID":"90b5e339-e754-42ff-96e1-e9d04d5570d0","Type":"ContainerStarted","Data":"f99979719b31a3f096bb0b4a3f651d7e29a2de4d9fb4d37ec6baacf3071fc84d"}
Jul 17 18:45:32.349997 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:32.349961    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c84488dd8-sb8dc" event={"ID":"90b5e339-e754-42ff-96e1-e9d04d5570d0","Type":"ContainerStarted","Data":"26c1eb779e0631200367b35d83ed127aa9a1265f3f330b12c98a4455c23dcd6c"}
Jul 17 18:45:32.350495 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:32.350069    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c84488dd8-sb8dc"
Jul 17 18:45:32.370120 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:32.370073    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7c84488dd8-sb8dc" podStartSLOduration=1.416396314 podStartE2EDuration="4.370058598s" totalImagesPullingTime="2.953662284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:32.36865398 +0000 UTC m=+742.810162670" watchObservedRunningTime="2026-07-17 18:45:32.370058598 +0000 UTC m=+742.811567295"
Jul 17 18:45:43.359461 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:45:43.359431    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c84488dd8-sb8dc"
Jul 17 18:46:04.477332 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:04.477285    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-c574cb76b-x9kng"]
Jul 17 18:46:04.480982 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:04.480963    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c574cb76b-x9kng"
Jul 17 18:46:04.484464 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:04.484443    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 17 18:46:04.491770 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:04.491747    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c574cb76b-x9kng"]
Jul 17 18:46:04.619615 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:04.619582    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69778d64-192f-4acb-a18c-9b849d7b00ce-maas-api-tls\") pod \"maas-api-c574cb76b-x9kng\" (UID: \"69778d64-192f-4acb-a18c-9b849d7b00ce\") " pod="odh-ai-gateway-infra/maas-api-c574cb76b-x9kng"
Jul 17 18:46:04.619796 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:04.619621    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69778d64-192f-4acb-a18c-9b849d7b00ce-openshift-service-ca\") pod \"maas-api-c574cb76b-x9kng\" (UID: \"69778d64-192f-4acb-a18c-9b849d7b00ce\") " pod="odh-ai-gateway-infra/maas-api-c574cb76b-x9kng"
Jul 17 18:46:04.619796 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:04.619761    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlb2d\" (UniqueName: \"kubernetes.io/projected/69778d64-192f-4acb-a18c-9b849d7b00ce-kube-api-access-rlb2d\") pod \"maas-api-c574cb76b-x9kng\" (UID: \"69778d64-192f-4acb-a18c-9b849d7b00ce\") " pod="odh-ai-gateway-infra/maas-api-c574cb76b-x9kng"
Jul 17 18:46:04.720902 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:04.720866    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlb2d\" (UniqueName: \"kubernetes.io/projected/69778d64-192f-4acb-a18c-9b849d7b00ce-kube-api-access-rlb2d\") pod \"maas-api-c574cb76b-x9kng\" (UID: \"69778d64-192f-4acb-a18c-9b849d7b00ce\") " pod="odh-ai-gateway-infra/maas-api-c574cb76b-x9kng"
Jul 17 18:46:04.721103 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:04.720924    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69778d64-192f-4acb-a18c-9b849d7b00ce-maas-api-tls\") pod \"maas-api-c574cb76b-x9kng\" (UID: \"69778d64-192f-4acb-a18c-9b849d7b00ce\") " pod="odh-ai-gateway-infra/maas-api-c574cb76b-x9kng"
Jul 17 18:46:04.721103 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:04.720938    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69778d64-192f-4acb-a18c-9b849d7b00ce-openshift-service-ca\") pod \"maas-api-c574cb76b-x9kng\" (UID: \"69778d64-192f-4acb-a18c-9b849d7b00ce\") " pod="odh-ai-gateway-infra/maas-api-c574cb76b-x9kng"
Jul 17 18:46:04.721690 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:04.721667    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69778d64-192f-4acb-a18c-9b849d7b00ce-openshift-service-ca\") pod \"maas-api-c574cb76b-x9kng\" (UID: \"69778d64-192f-4acb-a18c-9b849d7b00ce\") " pod="odh-ai-gateway-infra/maas-api-c574cb76b-x9kng"
Jul 17 18:46:04.723463 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:04.723445    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69778d64-192f-4acb-a18c-9b849d7b00ce-maas-api-tls\") pod \"maas-api-c574cb76b-x9kng\" (UID: \"69778d64-192f-4acb-a18c-9b849d7b00ce\") " pod="odh-ai-gateway-infra/maas-api-c574cb76b-x9kng"
Jul 17 18:46:04.734170 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:04.734114    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlb2d\" (UniqueName: \"kubernetes.io/projected/69778d64-192f-4acb-a18c-9b849d7b00ce-kube-api-access-rlb2d\") pod \"maas-api-c574cb76b-x9kng\" (UID: \"69778d64-192f-4acb-a18c-9b849d7b00ce\") " pod="odh-ai-gateway-infra/maas-api-c574cb76b-x9kng"
Jul 17 18:46:04.793198 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:04.793153    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c574cb76b-x9kng"
Jul 17 18:46:04.943248 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:04.943221    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c574cb76b-x9kng"]
Jul 17 18:46:04.945223 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:46:04.945192    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69778d64_192f_4acb_a18c_9b849d7b00ce.slice/crio-ab01357b346031eaa0c8f3586a10cb51272b9a3ac76dbf865ca916014023ae49 WatchSource:0}: Error finding container ab01357b346031eaa0c8f3586a10cb51272b9a3ac76dbf865ca916014023ae49: Status 404 returned error can't find the container with id ab01357b346031eaa0c8f3586a10cb51272b9a3ac76dbf865ca916014023ae49
Jul 17 18:46:05.528121 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:05.528088    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c574cb76b-x9kng" event={"ID":"69778d64-192f-4acb-a18c-9b849d7b00ce","Type":"ContainerStarted","Data":"ab01357b346031eaa0c8f3586a10cb51272b9a3ac76dbf865ca916014023ae49"}
Jul 17 18:46:07.540971 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:07.540936    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c574cb76b-x9kng" event={"ID":"69778d64-192f-4acb-a18c-9b849d7b00ce","Type":"ContainerStarted","Data":"2f9cbfb678986ffb7e6e6cdc40e87e186fd0488a45347a3f2cc005990350c2e4"}
Jul 17 18:46:07.541406 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:07.540997    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-c574cb76b-x9kng"
Jul 17 18:46:07.560655 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:07.560607    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-c574cb76b-x9kng" podStartSLOduration=1.68329434 podStartE2EDuration="3.560590531s" totalImagesPullingTime="1.877296191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:46:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:46:07.558938865 +0000 UTC m=+778.000447565" watchObservedRunningTime="2026-07-17 18:46:07.560590531 +0000 UTC m=+778.002099265"
Jul 17 18:46:13.551654 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:13.551626    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-c574cb76b-x9kng"
Jul 17 18:46:21.591706 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.591670    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"]
Jul 17 18:46:21.595445 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.595425    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:21.598230 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.598212    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 17 18:46:21.598348 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.598246    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 17 18:46:21.599610 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.599583    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-skht6\""
Jul 17 18:46:21.599841 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.599825    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 17 18:46:21.609584 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.609564    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"]
Jul 17 18:46:21.777281 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.777246    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ebe5a2-2fa1-449a-a873-b551f2cfcc7b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj\" (UID: \"67ebe5a2-2fa1-449a-a873-b551f2cfcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:21.777478 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.777288    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwg7t\" (UniqueName: \"kubernetes.io/projected/67ebe5a2-2fa1-449a-a873-b551f2cfcc7b-kube-api-access-zwg7t\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj\" (UID: \"67ebe5a2-2fa1-449a-a873-b551f2cfcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:21.777478 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.777411    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ebe5a2-2fa1-449a-a873-b551f2cfcc7b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj\" (UID: \"67ebe5a2-2fa1-449a-a873-b551f2cfcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:21.777478 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.777433    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ebe5a2-2fa1-449a-a873-b551f2cfcc7b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj\" (UID: \"67ebe5a2-2fa1-449a-a873-b551f2cfcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:21.777478 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.777447    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ebe5a2-2fa1-449a-a873-b551f2cfcc7b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj\" (UID: \"67ebe5a2-2fa1-449a-a873-b551f2cfcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:21.777478 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.777471    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ebe5a2-2fa1-449a-a873-b551f2cfcc7b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj\" (UID: \"67ebe5a2-2fa1-449a-a873-b551f2cfcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:21.878540 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.878461    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ebe5a2-2fa1-449a-a873-b551f2cfcc7b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj\" (UID: \"67ebe5a2-2fa1-449a-a873-b551f2cfcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:21.878540 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.878505    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwg7t\" (UniqueName: \"kubernetes.io/projected/67ebe5a2-2fa1-449a-a873-b551f2cfcc7b-kube-api-access-zwg7t\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj\" (UID: \"67ebe5a2-2fa1-449a-a873-b551f2cfcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:21.878758 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.878569    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ebe5a2-2fa1-449a-a873-b551f2cfcc7b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj\" (UID: \"67ebe5a2-2fa1-449a-a873-b551f2cfcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:21.878758 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.878596    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ebe5a2-2fa1-449a-a873-b551f2cfcc7b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj\" (UID: \"67ebe5a2-2fa1-449a-a873-b551f2cfcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:21.878758 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.878619    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ebe5a2-2fa1-449a-a873-b551f2cfcc7b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj\" (UID: \"67ebe5a2-2fa1-449a-a873-b551f2cfcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:21.878758 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.878655    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ebe5a2-2fa1-449a-a873-b551f2cfcc7b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj\" (UID: \"67ebe5a2-2fa1-449a-a873-b551f2cfcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:21.878970 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.878888    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ebe5a2-2fa1-449a-a873-b551f2cfcc7b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj\" (UID: \"67ebe5a2-2fa1-449a-a873-b551f2cfcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:21.878970 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.878910    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ebe5a2-2fa1-449a-a873-b551f2cfcc7b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj\" (UID: \"67ebe5a2-2fa1-449a-a873-b551f2cfcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:21.879069 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.879021    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ebe5a2-2fa1-449a-a873-b551f2cfcc7b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj\" (UID: \"67ebe5a2-2fa1-449a-a873-b551f2cfcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:21.880977 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.880957    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ebe5a2-2fa1-449a-a873-b551f2cfcc7b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj\" (UID: \"67ebe5a2-2fa1-449a-a873-b551f2cfcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:21.881380 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.881361    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ebe5a2-2fa1-449a-a873-b551f2cfcc7b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj\" (UID: \"67ebe5a2-2fa1-449a-a873-b551f2cfcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:21.886785 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.886761    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwg7t\" (UniqueName: \"kubernetes.io/projected/67ebe5a2-2fa1-449a-a873-b551f2cfcc7b-kube-api-access-zwg7t\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj\" (UID: \"67ebe5a2-2fa1-449a-a873-b551f2cfcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:21.906779 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:21.906758    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:22.039828 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:22.039804    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"]
Jul 17 18:46:22.041883 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:46:22.041858    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ebe5a2_2fa1_449a_a873_b551f2cfcc7b.slice/crio-ce031ab6ac1b648b8640ebddcc68103a37cd0fef736455453007feec864bb3c0 WatchSource:0}: Error finding container ce031ab6ac1b648b8640ebddcc68103a37cd0fef736455453007feec864bb3c0: Status 404 returned error can't find the container with id ce031ab6ac1b648b8640ebddcc68103a37cd0fef736455453007feec864bb3c0
Jul 17 18:46:22.623534 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:22.623466    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj" event={"ID":"67ebe5a2-2fa1-449a-a873-b551f2cfcc7b","Type":"ContainerStarted","Data":"ce031ab6ac1b648b8640ebddcc68103a37cd0fef736455453007feec864bb3c0"}
Jul 17 18:46:23.630397 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:23.630363    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj" event={"ID":"67ebe5a2-2fa1-449a-a873-b551f2cfcc7b","Type":"ContainerStarted","Data":"6a08b3f58484571485d9dcdfda4f7e35652ea832cd6ab7912896183fdfae417d"}
Jul 17 18:46:23.650985 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:23.650937    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj" podStartSLOduration=1.5619330200000001 podStartE2EDuration="2.650922234s" totalImagesPullingTime="1.088989214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:46:23.648534201 +0000 UTC m=+794.090042891" watchObservedRunningTime="2026-07-17 18:46:23.650922234 +0000 UTC m=+794.092430986"
Jul 17 18:46:31.907455 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:31.907376    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:31.907455 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:31.907436    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:31.920216 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:31.920188    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:32.689225 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:32.689193    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj"
Jul 17 18:46:44.798997 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.798967    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"]
Jul 17 18:46:44.803758 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.803736    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:44.806455 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.806437    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 17 18:46:44.812872 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.812852    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"]
Jul 17 18:46:44.845751 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.845723    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e6fdeec-4355-445a-9d68-aca800edc6e6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6\" (UID: \"3e6fdeec-4355-445a-9d68-aca800edc6e6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:44.845857 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.845758    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e6fdeec-4355-445a-9d68-aca800edc6e6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6\" (UID: \"3e6fdeec-4355-445a-9d68-aca800edc6e6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:44.845857 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.845793    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6fdeec-4355-445a-9d68-aca800edc6e6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6\" (UID: \"3e6fdeec-4355-445a-9d68-aca800edc6e6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:44.845857 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.845839    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vhwf\" (UniqueName: \"kubernetes.io/projected/3e6fdeec-4355-445a-9d68-aca800edc6e6-kube-api-access-5vhwf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6\" (UID: \"3e6fdeec-4355-445a-9d68-aca800edc6e6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:44.845959 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.845920    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e6fdeec-4355-445a-9d68-aca800edc6e6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6\" (UID: \"3e6fdeec-4355-445a-9d68-aca800edc6e6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:44.845959 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.845939    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6fdeec-4355-445a-9d68-aca800edc6e6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6\" (UID: \"3e6fdeec-4355-445a-9d68-aca800edc6e6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:44.946831 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.946805    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e6fdeec-4355-445a-9d68-aca800edc6e6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6\" (UID: \"3e6fdeec-4355-445a-9d68-aca800edc6e6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:44.946963 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.946837    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e6fdeec-4355-445a-9d68-aca800edc6e6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6\" (UID: \"3e6fdeec-4355-445a-9d68-aca800edc6e6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:44.946963 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.946858    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6fdeec-4355-445a-9d68-aca800edc6e6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6\" (UID: \"3e6fdeec-4355-445a-9d68-aca800edc6e6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:44.946963 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.946875    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vhwf\" (UniqueName: \"kubernetes.io/projected/3e6fdeec-4355-445a-9d68-aca800edc6e6-kube-api-access-5vhwf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6\" (UID: \"3e6fdeec-4355-445a-9d68-aca800edc6e6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:44.946963 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.946909    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e6fdeec-4355-445a-9d68-aca800edc6e6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6\" (UID: \"3e6fdeec-4355-445a-9d68-aca800edc6e6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:44.947166 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.947020    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6fdeec-4355-445a-9d68-aca800edc6e6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6\" (UID: \"3e6fdeec-4355-445a-9d68-aca800edc6e6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:44.947253 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.947233    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e6fdeec-4355-445a-9d68-aca800edc6e6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6\" (UID: \"3e6fdeec-4355-445a-9d68-aca800edc6e6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:44.947253 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.947246    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e6fdeec-4355-445a-9d68-aca800edc6e6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6\" (UID: \"3e6fdeec-4355-445a-9d68-aca800edc6e6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:44.947383 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.947334    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6fdeec-4355-445a-9d68-aca800edc6e6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6\" (UID: \"3e6fdeec-4355-445a-9d68-aca800edc6e6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:44.949256 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.949236    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e6fdeec-4355-445a-9d68-aca800edc6e6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6\" (UID: \"3e6fdeec-4355-445a-9d68-aca800edc6e6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:44.949673 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.949651    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6fdeec-4355-445a-9d68-aca800edc6e6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6\" (UID: \"3e6fdeec-4355-445a-9d68-aca800edc6e6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:44.956139 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:44.956122    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vhwf\" (UniqueName: \"kubernetes.io/projected/3e6fdeec-4355-445a-9d68-aca800edc6e6-kube-api-access-5vhwf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6\" (UID: \"3e6fdeec-4355-445a-9d68-aca800edc6e6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:45.114704 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:45.114659    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:45.249965 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:45.249937    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"]
Jul 17 18:46:45.253440 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:46:45.253411    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6fdeec_4355_445a_9d68_aca800edc6e6.slice/crio-4554c266d816f0c817a4d1749d8f990ed410fcc7becb6f1201117c20854778f7 WatchSource:0}: Error finding container 4554c266d816f0c817a4d1749d8f990ed410fcc7becb6f1201117c20854778f7: Status 404 returned error can't find the container with id 4554c266d816f0c817a4d1749d8f990ed410fcc7becb6f1201117c20854778f7
Jul 17 18:46:45.746396 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:45.746296    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6" event={"ID":"3e6fdeec-4355-445a-9d68-aca800edc6e6","Type":"ContainerStarted","Data":"65f2e75bc34aec6f2f3d6f4049d189f4a7b14c7683664c7f9e796441d9b6dd7b"}
Jul 17 18:46:45.746396 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:45.746367    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6" event={"ID":"3e6fdeec-4355-445a-9d68-aca800edc6e6","Type":"ContainerStarted","Data":"4554c266d816f0c817a4d1749d8f990ed410fcc7becb6f1201117c20854778f7"}
Jul 17 18:46:45.766570 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:45.766525    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6" podStartSLOduration=1.5905233669999999 podStartE2EDuration="1.766510466s" totalImagesPullingTime="175.987099ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:46:45.764329289 +0000 UTC m=+816.205837971" watchObservedRunningTime="2026-07-17 18:46:45.766510466 +0000 UTC m=+816.208019154"
Jul 17 18:46:55.115370 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:55.115335    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:55.115839 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:55.115403    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:55.127681 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:55.127660    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:46:55.809320 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:46:55.809289    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6"
Jul 17 18:48:10.098567 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:10.098534    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 18:48:10.100628 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:10.100606    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 18:48:10.109980 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:10.109957    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 18:48:10.110879 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:10.110855    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 18:48:56.662604 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:56.662565    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c84488dd8-sb8dc"]
Jul 17 18:48:56.663137 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:56.662820    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7c84488dd8-sb8dc" podUID="90b5e339-e754-42ff-96e1-e9d04d5570d0" containerName="manager" containerID="cri-o://26c1eb779e0631200367b35d83ed127aa9a1265f3f330b12c98a4455c23dcd6c" gracePeriod=30
Jul 17 18:48:56.913487 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:56.913428    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c84488dd8-sb8dc"
Jul 17 18:48:56.982855 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:56.982821    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90b5e339-e754-42ff-96e1-e9d04d5570d0-webhook-cert\") pod \"90b5e339-e754-42ff-96e1-e9d04d5570d0\" (UID: \"90b5e339-e754-42ff-96e1-e9d04d5570d0\") "
Jul 17 18:48:56.983027 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:56.982900    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdq8f\" (UniqueName: \"kubernetes.io/projected/90b5e339-e754-42ff-96e1-e9d04d5570d0-kube-api-access-wdq8f\") pod \"90b5e339-e754-42ff-96e1-e9d04d5570d0\" (UID: \"90b5e339-e754-42ff-96e1-e9d04d5570d0\") "
Jul 17 18:48:56.985188 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:56.985159    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90b5e339-e754-42ff-96e1-e9d04d5570d0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "90b5e339-e754-42ff-96e1-e9d04d5570d0" (UID: "90b5e339-e754-42ff-96e1-e9d04d5570d0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:48:56.985336 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:56.985220    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b5e339-e754-42ff-96e1-e9d04d5570d0-kube-api-access-wdq8f" (OuterVolumeSpecName: "kube-api-access-wdq8f") pod "90b5e339-e754-42ff-96e1-e9d04d5570d0" (UID: "90b5e339-e754-42ff-96e1-e9d04d5570d0"). InnerVolumeSpecName "kube-api-access-wdq8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:48:57.083997 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:57.083963    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdq8f\" (UniqueName: \"kubernetes.io/projected/90b5e339-e754-42ff-96e1-e9d04d5570d0-kube-api-access-wdq8f\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:48:57.083997 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:57.083991    2559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90b5e339-e754-42ff-96e1-e9d04d5570d0-webhook-cert\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:48:57.464487 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:57.464450    2559 generic.go:356] "Generic (PLEG): container finished" podID="90b5e339-e754-42ff-96e1-e9d04d5570d0" containerID="26c1eb779e0631200367b35d83ed127aa9a1265f3f330b12c98a4455c23dcd6c" exitCode=0
Jul 17 18:48:57.464487 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:57.464490    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c84488dd8-sb8dc" event={"ID":"90b5e339-e754-42ff-96e1-e9d04d5570d0","Type":"ContainerDied","Data":"26c1eb779e0631200367b35d83ed127aa9a1265f3f330b12c98a4455c23dcd6c"}
Jul 17 18:48:57.464745 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:57.464504    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c84488dd8-sb8dc"
Jul 17 18:48:57.464745 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:57.464532    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c84488dd8-sb8dc" event={"ID":"90b5e339-e754-42ff-96e1-e9d04d5570d0","Type":"ContainerDied","Data":"f99979719b31a3f096bb0b4a3f651d7e29a2de4d9fb4d37ec6baacf3071fc84d"}
Jul 17 18:48:57.464745 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:57.464553    2559 scope.go:117] "RemoveContainer" containerID="26c1eb779e0631200367b35d83ed127aa9a1265f3f330b12c98a4455c23dcd6c"
Jul 17 18:48:57.474336 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:57.474297    2559 scope.go:117] "RemoveContainer" containerID="26c1eb779e0631200367b35d83ed127aa9a1265f3f330b12c98a4455c23dcd6c"
Jul 17 18:48:57.474616 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:48:57.474598    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c1eb779e0631200367b35d83ed127aa9a1265f3f330b12c98a4455c23dcd6c\": container with ID starting with 26c1eb779e0631200367b35d83ed127aa9a1265f3f330b12c98a4455c23dcd6c not found: ID does not exist" containerID="26c1eb779e0631200367b35d83ed127aa9a1265f3f330b12c98a4455c23dcd6c"
Jul 17 18:48:57.474676 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:57.474625    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c1eb779e0631200367b35d83ed127aa9a1265f3f330b12c98a4455c23dcd6c"} err="failed to get container status \"26c1eb779e0631200367b35d83ed127aa9a1265f3f330b12c98a4455c23dcd6c\": rpc error: code = NotFound desc = could not find container \"26c1eb779e0631200367b35d83ed127aa9a1265f3f330b12c98a4455c23dcd6c\": container with ID starting with 26c1eb779e0631200367b35d83ed127aa9a1265f3f330b12c98a4455c23dcd6c not found: ID does not exist"
Jul 17 18:48:57.486202 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:57.486174    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c84488dd8-sb8dc"]
Jul 17 18:48:57.489657 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:57.489636    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7c84488dd8-sb8dc"]
Jul 17 18:48:58.126083 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:48:58.126048    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90b5e339-e754-42ff-96e1-e9d04d5570d0" path="/var/lib/kubelet/pods/90b5e339-e754-42ff-96e1-e9d04d5570d0/volumes"
Jul 17 18:49:58.296460 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:58.296378    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c84488dd8-qg7wf"]
Jul 17 18:49:58.297004 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:58.296945    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90b5e339-e754-42ff-96e1-e9d04d5570d0" containerName="manager"
Jul 17 18:49:58.297004 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:58.296964    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="90b5e339-e754-42ff-96e1-e9d04d5570d0" containerName="manager"
Jul 17 18:49:58.297120 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:58.297064    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="90b5e339-e754-42ff-96e1-e9d04d5570d0" containerName="manager"
Jul 17 18:49:58.300127 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:58.300107    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c84488dd8-qg7wf"
Jul 17 18:49:58.303723 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:58.303702    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 17 18:49:58.303723 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:58.303718    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 17 18:49:58.303908 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:58.303743    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8hd7g\""
Jul 17 18:49:58.310026 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:58.310000    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c84488dd8-qg7wf"]
Jul 17 18:49:58.419118 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:58.419092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sd25\" (UniqueName: \"kubernetes.io/projected/ba6abe08-7791-4314-81fa-114e7313117c-kube-api-access-5sd25\") pod \"maas-controller-7c84488dd8-qg7wf\" (UID: \"ba6abe08-7791-4314-81fa-114e7313117c\") " pod="opendatahub/maas-controller-7c84488dd8-qg7wf"
Jul 17 18:49:58.419262 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:58.419149    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba6abe08-7791-4314-81fa-114e7313117c-webhook-cert\") pod \"maas-controller-7c84488dd8-qg7wf\" (UID: \"ba6abe08-7791-4314-81fa-114e7313117c\") " pod="opendatahub/maas-controller-7c84488dd8-qg7wf"
Jul 17 18:49:58.520458 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:58.520428    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba6abe08-7791-4314-81fa-114e7313117c-webhook-cert\") pod \"maas-controller-7c84488dd8-qg7wf\" (UID: \"ba6abe08-7791-4314-81fa-114e7313117c\") " pod="opendatahub/maas-controller-7c84488dd8-qg7wf"
Jul 17 18:49:58.520587 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:58.520494    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sd25\" (UniqueName: \"kubernetes.io/projected/ba6abe08-7791-4314-81fa-114e7313117c-kube-api-access-5sd25\") pod \"maas-controller-7c84488dd8-qg7wf\" (UID: \"ba6abe08-7791-4314-81fa-114e7313117c\") " pod="opendatahub/maas-controller-7c84488dd8-qg7wf"
Jul 17 18:49:58.523013 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:58.522992    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba6abe08-7791-4314-81fa-114e7313117c-webhook-cert\") pod \"maas-controller-7c84488dd8-qg7wf\" (UID: \"ba6abe08-7791-4314-81fa-114e7313117c\") " pod="opendatahub/maas-controller-7c84488dd8-qg7wf"
Jul 17 18:49:58.529077 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:58.529052    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sd25\" (UniqueName: \"kubernetes.io/projected/ba6abe08-7791-4314-81fa-114e7313117c-kube-api-access-5sd25\") pod \"maas-controller-7c84488dd8-qg7wf\" (UID: \"ba6abe08-7791-4314-81fa-114e7313117c\") " pod="opendatahub/maas-controller-7c84488dd8-qg7wf"
Jul 17 18:49:58.610836 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:58.610812    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c84488dd8-qg7wf"
Jul 17 18:49:58.744911 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:58.744882    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c84488dd8-qg7wf"]
Jul 17 18:49:58.746139 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:49:58.746114    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6abe08_7791_4314_81fa_114e7313117c.slice/crio-e72bde3261f7aaf7b613304be545f1b04341b7b480e05cf6f69bf86cdec0183f WatchSource:0}: Error finding container e72bde3261f7aaf7b613304be545f1b04341b7b480e05cf6f69bf86cdec0183f: Status 404 returned error can't find the container with id e72bde3261f7aaf7b613304be545f1b04341b7b480e05cf6f69bf86cdec0183f
Jul 17 18:49:58.747469 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:58.747451    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 18:49:58.798232 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:58.798202    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c84488dd8-qg7wf" event={"ID":"ba6abe08-7791-4314-81fa-114e7313117c","Type":"ContainerStarted","Data":"e72bde3261f7aaf7b613304be545f1b04341b7b480e05cf6f69bf86cdec0183f"}
Jul 17 18:49:59.804774 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:59.804730    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c84488dd8-qg7wf" event={"ID":"ba6abe08-7791-4314-81fa-114e7313117c","Type":"ContainerStarted","Data":"207ed0e41b679009ec22f4c172f96fc8adc9e05745e0cead2492ae6a091f0beb"}
Jul 17 18:49:59.805166 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:59.804791    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c84488dd8-qg7wf"
Jul 17 18:49:59.826540 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:49:59.826494    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7c84488dd8-qg7wf" podStartSLOduration=1.388833956 podStartE2EDuration="1.826479619s" totalImagesPullingTime="437.645663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:49:59.824361575 +0000 UTC m=+1010.265870264" watchObservedRunningTime="2026-07-17 18:49:59.826479619 +0000 UTC m=+1010.267988319"
Jul 17 18:50:10.812084 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:10.812055    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c84488dd8-qg7wf"
Jul 17 18:50:15.442502 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:15.442471    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c84488dd8-qg7wf"]
Jul 17 18:50:15.442907 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:15.442696    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7c84488dd8-qg7wf" podUID="ba6abe08-7791-4314-81fa-114e7313117c" containerName="manager" containerID="cri-o://207ed0e41b679009ec22f4c172f96fc8adc9e05745e0cead2492ae6a091f0beb" gracePeriod=30
Jul 17 18:50:15.693354 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:15.693281    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c84488dd8-qg7wf"
Jul 17 18:50:15.865171 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:15.865139    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba6abe08-7791-4314-81fa-114e7313117c-webhook-cert\") pod \"ba6abe08-7791-4314-81fa-114e7313117c\" (UID: \"ba6abe08-7791-4314-81fa-114e7313117c\") "
Jul 17 18:50:15.865364 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:15.865210    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sd25\" (UniqueName: \"kubernetes.io/projected/ba6abe08-7791-4314-81fa-114e7313117c-kube-api-access-5sd25\") pod \"ba6abe08-7791-4314-81fa-114e7313117c\" (UID: \"ba6abe08-7791-4314-81fa-114e7313117c\") "
Jul 17 18:50:15.867296 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:15.867268    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6abe08-7791-4314-81fa-114e7313117c-kube-api-access-5sd25" (OuterVolumeSpecName: "kube-api-access-5sd25") pod "ba6abe08-7791-4314-81fa-114e7313117c" (UID: "ba6abe08-7791-4314-81fa-114e7313117c"). InnerVolumeSpecName "kube-api-access-5sd25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:50:15.867413 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:15.867268    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba6abe08-7791-4314-81fa-114e7313117c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ba6abe08-7791-4314-81fa-114e7313117c" (UID: "ba6abe08-7791-4314-81fa-114e7313117c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:50:15.895880 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:15.895855    2559 generic.go:356] "Generic (PLEG): container finished" podID="ba6abe08-7791-4314-81fa-114e7313117c" containerID="207ed0e41b679009ec22f4c172f96fc8adc9e05745e0cead2492ae6a091f0beb" exitCode=0
Jul 17 18:50:15.895970 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:15.895903    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c84488dd8-qg7wf"
Jul 17 18:50:15.895970 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:15.895945    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c84488dd8-qg7wf" event={"ID":"ba6abe08-7791-4314-81fa-114e7313117c","Type":"ContainerDied","Data":"207ed0e41b679009ec22f4c172f96fc8adc9e05745e0cead2492ae6a091f0beb"}
Jul 17 18:50:15.896135 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:15.896000    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c84488dd8-qg7wf" event={"ID":"ba6abe08-7791-4314-81fa-114e7313117c","Type":"ContainerDied","Data":"e72bde3261f7aaf7b613304be545f1b04341b7b480e05cf6f69bf86cdec0183f"}
Jul 17 18:50:15.896135 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:15.896028    2559 scope.go:117] "RemoveContainer" containerID="207ed0e41b679009ec22f4c172f96fc8adc9e05745e0cead2492ae6a091f0beb"
Jul 17 18:50:15.905401 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:15.905386    2559 scope.go:117] "RemoveContainer" containerID="207ed0e41b679009ec22f4c172f96fc8adc9e05745e0cead2492ae6a091f0beb"
Jul 17 18:50:15.905656 ip-10-0-138-51 kubenswrapper[2559]: E0717 18:50:15.905640    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"207ed0e41b679009ec22f4c172f96fc8adc9e05745e0cead2492ae6a091f0beb\": container with ID starting with 207ed0e41b679009ec22f4c172f96fc8adc9e05745e0cead2492ae6a091f0beb not found: ID does not exist" containerID="207ed0e41b679009ec22f4c172f96fc8adc9e05745e0cead2492ae6a091f0beb"
Jul 17 18:50:15.905698 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:15.905665    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"207ed0e41b679009ec22f4c172f96fc8adc9e05745e0cead2492ae6a091f0beb"} err="failed to get container status \"207ed0e41b679009ec22f4c172f96fc8adc9e05745e0cead2492ae6a091f0beb\": rpc error: code = NotFound desc = could not find container \"207ed0e41b679009ec22f4c172f96fc8adc9e05745e0cead2492ae6a091f0beb\": container with ID starting with 207ed0e41b679009ec22f4c172f96fc8adc9e05745e0cead2492ae6a091f0beb not found: ID does not exist"
Jul 17 18:50:15.932253 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:15.932228    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c84488dd8-qg7wf"]
Jul 17 18:50:15.950139 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:15.950088    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7c84488dd8-qg7wf"]
Jul 17 18:50:15.966346 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:15.966326    2559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba6abe08-7791-4314-81fa-114e7313117c-webhook-cert\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:50:15.966433 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:15.966348    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sd25\" (UniqueName: \"kubernetes.io/projected/ba6abe08-7791-4314-81fa-114e7313117c-kube-api-access-5sd25\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 18:50:16.125655 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:16.125629    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba6abe08-7791-4314-81fa-114e7313117c" path="/var/lib/kubelet/pods/ba6abe08-7791-4314-81fa-114e7313117c/volumes"
Jul 17 18:50:16.719905 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:16.719873    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c84488dd8-2nhq2"]
Jul 17 18:50:16.720263 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:16.720225    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba6abe08-7791-4314-81fa-114e7313117c" containerName="manager"
Jul 17 18:50:16.720263 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:16.720235    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba6abe08-7791-4314-81fa-114e7313117c" containerName="manager"
Jul 17 18:50:16.720371 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:16.720331    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba6abe08-7791-4314-81fa-114e7313117c" containerName="manager"
Jul 17 18:50:16.724592 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:16.724573    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c84488dd8-2nhq2"
Jul 17 18:50:16.727933 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:16.727907    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 17 18:50:16.728050 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:16.727998    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 17 18:50:16.728050 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:16.728010    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8hd7g\""
Jul 17 18:50:16.733024 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:16.732966    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c84488dd8-2nhq2"]
Jul 17 18:50:16.773858 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:16.773831    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94530b49-743c-4bac-812f-897823b999b2-webhook-cert\") pod \"maas-controller-7c84488dd8-2nhq2\" (UID: \"94530b49-743c-4bac-812f-897823b999b2\") " pod="opendatahub/maas-controller-7c84488dd8-2nhq2"
Jul 17 18:50:16.773989 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:16.773867    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shjbs\" (UniqueName: \"kubernetes.io/projected/94530b49-743c-4bac-812f-897823b999b2-kube-api-access-shjbs\") pod \"maas-controller-7c84488dd8-2nhq2\" (UID: \"94530b49-743c-4bac-812f-897823b999b2\") " pod="opendatahub/maas-controller-7c84488dd8-2nhq2"
Jul 17 18:50:16.874635 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:16.874610    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94530b49-743c-4bac-812f-897823b999b2-webhook-cert\") pod \"maas-controller-7c84488dd8-2nhq2\" (UID: \"94530b49-743c-4bac-812f-897823b999b2\") " pod="opendatahub/maas-controller-7c84488dd8-2nhq2"
Jul 17 18:50:16.874772 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:16.874648    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shjbs\" (UniqueName: \"kubernetes.io/projected/94530b49-743c-4bac-812f-897823b999b2-kube-api-access-shjbs\") pod \"maas-controller-7c84488dd8-2nhq2\" (UID: \"94530b49-743c-4bac-812f-897823b999b2\") " pod="opendatahub/maas-controller-7c84488dd8-2nhq2"
Jul 17 18:50:16.877162 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:16.877133    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94530b49-743c-4bac-812f-897823b999b2-webhook-cert\") pod \"maas-controller-7c84488dd8-2nhq2\" (UID: \"94530b49-743c-4bac-812f-897823b999b2\") " pod="opendatahub/maas-controller-7c84488dd8-2nhq2"
Jul 17 18:50:16.884848 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:16.884828    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shjbs\" (UniqueName: \"kubernetes.io/projected/94530b49-743c-4bac-812f-897823b999b2-kube-api-access-shjbs\") pod \"maas-controller-7c84488dd8-2nhq2\" (UID: \"94530b49-743c-4bac-812f-897823b999b2\") " pod="opendatahub/maas-controller-7c84488dd8-2nhq2"
Jul 17 18:50:17.037026 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:17.036949    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c84488dd8-2nhq2"
Jul 17 18:50:17.179208 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:17.179187    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c84488dd8-2nhq2"]
Jul 17 18:50:17.180136 ip-10-0-138-51 kubenswrapper[2559]: W0717 18:50:17.180113    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94530b49_743c_4bac_812f_897823b999b2.slice/crio-d44fa9a56d6a3323770e1aafc957d4318b0cd794f042260fda2d8b8aa7326ad8 WatchSource:0}: Error finding container d44fa9a56d6a3323770e1aafc957d4318b0cd794f042260fda2d8b8aa7326ad8: Status 404 returned error can't find the container with id d44fa9a56d6a3323770e1aafc957d4318b0cd794f042260fda2d8b8aa7326ad8
Jul 17 18:50:17.915514 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:17.915481    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c84488dd8-2nhq2" event={"ID":"94530b49-743c-4bac-812f-897823b999b2","Type":"ContainerStarted","Data":"d44fa9a56d6a3323770e1aafc957d4318b0cd794f042260fda2d8b8aa7326ad8"}
Jul 17 18:50:18.922369 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:18.922331    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c84488dd8-2nhq2" event={"ID":"94530b49-743c-4bac-812f-897823b999b2","Type":"ContainerStarted","Data":"688aba511b0318e15b18a1ec854dd27cc56029f840a35711e202adfe78ee0289"}
Jul 17 18:50:18.922831 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:18.922443    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c84488dd8-2nhq2"
Jul 17 18:50:18.939740 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:18.939689    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7c84488dd8-2nhq2" podStartSLOduration=2.254801315 podStartE2EDuration="2.939672496s" totalImagesPullingTime="684.871181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:50:18.938934776 +0000 UTC m=+1029.380443465" watchObservedRunningTime="2026-07-17 18:50:18.939672496 +0000 UTC m=+1029.381181187"
Jul 17 18:50:29.929737 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:50:29.929709    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c84488dd8-2nhq2"
Jul 17 18:53:10.142756 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:53:10.142725    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 18:53:10.144573 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:53:10.144552    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 18:53:10.150118 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:53:10.150100    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 18:53:10.151906 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:53:10.151889    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 18:58:10.182526 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:58:10.182495    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 18:58:10.188247 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:58:10.188222    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 18:58:10.191255 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:58:10.191234    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 18:58:10.196229 ip-10-0-138-51 kubenswrapper[2559]: I0717 18:58:10.196211    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 19:00:00.148838 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:00:00.148761    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738580-5ppb8"]
Jul 17 19:00:00.152225 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:00:00.152210    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738580-5ppb8"
Jul 17 19:00:00.155231 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:00:00.155208    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:00:00.161664 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:00:00.161645    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738580-5ppb8"
Jul 17 19:00:00.167842 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:00:00.167820    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738580-5ppb8"]
Jul 17 19:00:00.298385 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:00:00.298360    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738580-5ppb8"]
Jul 17 19:00:00.300446 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:00:00.300419    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e69000d_50b6_4f0e_8e42_d56be0992b71.slice/crio-b3d30d0a705487fa68aefceed5e766f38c5d6a0b76a7e9c44874e29e5947da09 WatchSource:0}: Error finding container b3d30d0a705487fa68aefceed5e766f38c5d6a0b76a7e9c44874e29e5947da09: Status 404 returned error can't find the container with id b3d30d0a705487fa68aefceed5e766f38c5d6a0b76a7e9c44874e29e5947da09
Jul 17 19:00:00.302786 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:00:00.302771    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 19:00:01.062812 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:00:01.062781    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738580-5ppb8" event={"ID":"7e69000d-50b6-4f0e-8e42-d56be0992b71","Type":"ContainerStarted","Data":"b3d30d0a705487fa68aefceed5e766f38c5d6a0b76a7e9c44874e29e5947da09"}
Jul 17 19:00:05.088230 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:00:05.088197    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738580-5ppb8" event={"ID":"7e69000d-50b6-4f0e-8e42-d56be0992b71","Type":"ContainerStarted","Data":"d81a3ed5975ea7f92d29f145d701aa08fbfc2f6c8dabe22b5911abd4d3cfab56"}
Jul 17 19:00:05.105112 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:00:05.105064    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738580-5ppb8" podStartSLOduration=0.682927037 podStartE2EDuration="5.105049857s" totalImagesPullingTime="4.42212282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:00:05.101917255 +0000 UTC m=+1615.543425941" watchObservedRunningTime="2026-07-17 19:00:05.105049857 +0000 UTC m=+1615.546558582"
Jul 17 19:00:06.094795 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:00:06.094757    2559 generic.go:356] "Generic (PLEG): container finished" podID="7e69000d-50b6-4f0e-8e42-d56be0992b71" containerID="d81a3ed5975ea7f92d29f145d701aa08fbfc2f6c8dabe22b5911abd4d3cfab56" exitCode=0
Jul 17 19:00:06.095146 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:00:06.094842    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738580-5ppb8" event={"ID":"7e69000d-50b6-4f0e-8e42-d56be0992b71","Type":"ContainerDied","Data":"d81a3ed5975ea7f92d29f145d701aa08fbfc2f6c8dabe22b5911abd4d3cfab56"}
Jul 17 19:00:07.234800 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:00:07.234778    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738580-5ppb8"
Jul 17 19:00:08.106955 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:00:08.106922    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738580-5ppb8" event={"ID":"7e69000d-50b6-4f0e-8e42-d56be0992b71","Type":"ContainerDied","Data":"b3d30d0a705487fa68aefceed5e766f38c5d6a0b76a7e9c44874e29e5947da09"}
Jul 17 19:00:08.106955 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:00:08.106948    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738580-5ppb8"
Jul 17 19:00:08.107162 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:00:08.106993    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3d30d0a705487fa68aefceed5e766f38c5d6a0b76a7e9c44874e29e5947da09"
Jul 17 19:02:40.072546 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:40.072512    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"]
Jul 17 19:02:40.072942 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:40.072926    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e69000d-50b6-4f0e-8e42-d56be0992b71" containerName="cleanup"
Jul 17 19:02:40.072942 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:40.072938    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e69000d-50b6-4f0e-8e42-d56be0992b71" containerName="cleanup"
Jul 17 19:02:40.073013 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:40.073004    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e69000d-50b6-4f0e-8e42-d56be0992b71" containerName="cleanup"
Jul 17 19:02:40.075092 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:40.075069    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:02:40.077516 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:40.077495    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7dc4fd53-serving-cert\""
Jul 17 19:02:40.077678 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:40.077659    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sbjr2\""
Jul 17 19:02:40.087216 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:40.087190    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"]
Jul 17 19:02:40.150588 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:40.150560    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe06fcb-35bf-44df-b010-c65504e8a47f-openshift-service-ca\") pod \"maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c\" (UID: \"abe06fcb-35bf-44df-b010-c65504e8a47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:02:40.150730 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:40.150610    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe06fcb-35bf-44df-b010-c65504e8a47f-maas-api-tls\") pod \"maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c\" (UID: \"abe06fcb-35bf-44df-b010-c65504e8a47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:02:40.150730 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:40.150692    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqnnr\" (UniqueName: \"kubernetes.io/projected/abe06fcb-35bf-44df-b010-c65504e8a47f-kube-api-access-lqnnr\") pod \"maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c\" (UID: \"abe06fcb-35bf-44df-b010-c65504e8a47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:02:40.251566 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:40.251535    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqnnr\" (UniqueName: \"kubernetes.io/projected/abe06fcb-35bf-44df-b010-c65504e8a47f-kube-api-access-lqnnr\") pod \"maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c\" (UID: \"abe06fcb-35bf-44df-b010-c65504e8a47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:02:40.251698 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:40.251588    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe06fcb-35bf-44df-b010-c65504e8a47f-openshift-service-ca\") pod \"maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c\" (UID: \"abe06fcb-35bf-44df-b010-c65504e8a47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:02:40.251698 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:40.251623    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe06fcb-35bf-44df-b010-c65504e8a47f-maas-api-tls\") pod \"maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c\" (UID: \"abe06fcb-35bf-44df-b010-c65504e8a47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:02:40.251813 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:02:40.251727    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-serving-cert: secret "maas-api-e2e-ait-7dc4fd53-serving-cert" not found
Jul 17 19:02:40.251813 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:02:40.251797    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abe06fcb-35bf-44df-b010-c65504e8a47f-maas-api-tls podName:abe06fcb-35bf-44df-b010-c65504e8a47f nodeName:}" failed. No retries permitted until 2026-07-17 19:02:40.75177767 +0000 UTC m=+1771.193286351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/abe06fcb-35bf-44df-b010-c65504e8a47f-maas-api-tls") pod "maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" (UID: "abe06fcb-35bf-44df-b010-c65504e8a47f") : secret "maas-api-e2e-ait-7dc4fd53-serving-cert" not found
Jul 17 19:02:40.252344 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:40.252297    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe06fcb-35bf-44df-b010-c65504e8a47f-openshift-service-ca\") pod \"maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c\" (UID: \"abe06fcb-35bf-44df-b010-c65504e8a47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:02:40.261574 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:40.261549    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqnnr\" (UniqueName: \"kubernetes.io/projected/abe06fcb-35bf-44df-b010-c65504e8a47f-kube-api-access-lqnnr\") pod \"maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c\" (UID: \"abe06fcb-35bf-44df-b010-c65504e8a47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:02:40.754923 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:40.754886    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe06fcb-35bf-44df-b010-c65504e8a47f-maas-api-tls\") pod \"maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c\" (UID: \"abe06fcb-35bf-44df-b010-c65504e8a47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:02:40.757466 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:40.757442    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe06fcb-35bf-44df-b010-c65504e8a47f-maas-api-tls\") pod \"maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c\" (UID: \"abe06fcb-35bf-44df-b010-c65504e8a47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:02:40.988036 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:40.988004    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:02:41.128784 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:41.128760    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"]
Jul 17 19:02:41.939816 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:41.939776    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" event={"ID":"abe06fcb-35bf-44df-b010-c65504e8a47f","Type":"ContainerStarted","Data":"715d4dec095e223a6026adb23b266f8a4cd85f5a5149f86a856e652a09afee29"}
Jul 17 19:02:41.939983 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:41.939834    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" event={"ID":"abe06fcb-35bf-44df-b010-c65504e8a47f","Type":"ContainerStarted","Data":"1a1b0f51f0cdbb21f7ede65f866fe4ee7142bd40f40417bde9c2428654dd135a"}
Jul 17 19:02:41.939983 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:41.939854    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:02:41.974000 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:41.973957    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podStartSLOduration=1.351323093 podStartE2EDuration="1.973942184s" totalImagesPullingTime="622.619091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:02:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:02:41.971084024 +0000 UTC m=+1772.412592712" watchObservedRunningTime="2026-07-17 19:02:41.973942184 +0000 UTC m=+1772.415450871"
Jul 17 19:02:42.946513 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:42.946476    2559 generic.go:356] "Generic (PLEG): container finished" podID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerID="715d4dec095e223a6026adb23b266f8a4cd85f5a5149f86a856e652a09afee29" exitCode=1
Jul 17 19:02:42.946879 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:42.946541    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" event={"ID":"abe06fcb-35bf-44df-b010-c65504e8a47f","Type":"ContainerDied","Data":"715d4dec095e223a6026adb23b266f8a4cd85f5a5149f86a856e652a09afee29"}
Jul 17 19:02:42.946879 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:42.946763    2559 scope.go:117] "RemoveContainer" containerID="715d4dec095e223a6026adb23b266f8a4cd85f5a5149f86a856e652a09afee29"
Jul 17 19:02:43.953698 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:43.953667    2559 generic.go:356] "Generic (PLEG): container finished" podID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerID="215b6c162e7b8fb17f5c4879ecdd34a8d573621d2f1c210f0119af0d71ad3360" exitCode=1
Jul 17 19:02:43.954077 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:43.953747    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" event={"ID":"abe06fcb-35bf-44df-b010-c65504e8a47f","Type":"ContainerDied","Data":"215b6c162e7b8fb17f5c4879ecdd34a8d573621d2f1c210f0119af0d71ad3360"}
Jul 17 19:02:43.954077 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:43.953809    2559 scope.go:117] "RemoveContainer" containerID="715d4dec095e223a6026adb23b266f8a4cd85f5a5149f86a856e652a09afee29"
Jul 17 19:02:43.954077 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:43.954054    2559 scope.go:117] "RemoveContainer" containerID="215b6c162e7b8fb17f5c4879ecdd34a8d573621d2f1c210f0119af0d71ad3360"
Jul 17 19:02:43.954278 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:02:43.954256    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:02:44.924801 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:44.924762    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j"]
Jul 17 19:02:44.928508 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:44.928484    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j"
Jul 17 19:02:44.931093 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:44.931068    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:02:44.937583 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:44.937112    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j"]
Jul 17 19:02:44.961571 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:44.961536    2559 scope.go:117] "RemoveContainer" containerID="215b6c162e7b8fb17f5c4879ecdd34a8d573621d2f1c210f0119af0d71ad3360"
Jul 17 19:02:44.961996 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:02:44.961823    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:02:44.994644 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:44.994613    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a359e9b-33ff-4bbe-87ab-26e8320533d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j\" (UID: \"5a359e9b-33ff-4bbe-87ab-26e8320533d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j"
Jul 17 19:02:45.096077 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:45.096046    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a359e9b-33ff-4bbe-87ab-26e8320533d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j\" (UID: \"5a359e9b-33ff-4bbe-87ab-26e8320533d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j"
Jul 17 19:02:45.096666 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:45.096647    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a359e9b-33ff-4bbe-87ab-26e8320533d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j\" (UID: \"5a359e9b-33ff-4bbe-87ab-26e8320533d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j"
Jul 17 19:02:45.242633 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:45.242563    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j"
Jul 17 19:02:45.397169 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:45.397132    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j"]
Jul 17 19:02:45.403774 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:02:45.403750    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a359e9b_33ff_4bbe_87ab_26e8320533d8.slice/crio-4417b55ba09c8acef59da9690cead6a8eb9722f7ff40f87f102c796e9fa4f809 WatchSource:0}: Error finding container 4417b55ba09c8acef59da9690cead6a8eb9722f7ff40f87f102c796e9fa4f809: Status 404 returned error can't find the container with id 4417b55ba09c8acef59da9690cead6a8eb9722f7ff40f87f102c796e9fa4f809
Jul 17 19:02:45.968338 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:45.968233    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j" event={"ID":"5a359e9b-33ff-4bbe-87ab-26e8320533d8","Type":"ContainerStarted","Data":"4cdfb4e5eebe9c4cb282d81c00a0ee637383083c0bfe228ac9516a197b5bee48"}
Jul 17 19:02:45.968338 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:45.968283    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j" event={"ID":"5a359e9b-33ff-4bbe-87ab-26e8320533d8","Type":"ContainerStarted","Data":"4417b55ba09c8acef59da9690cead6a8eb9722f7ff40f87f102c796e9fa4f809"}
Jul 17 19:02:45.985873 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:45.985828    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j" podStartSLOduration=1.985814168 podStartE2EDuration="1.985814168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:02:45.984422915 +0000 UTC m=+1776.425931602" watchObservedRunningTime="2026-07-17 19:02:45.985814168 +0000 UTC m=+1776.427322857"
Jul 17 19:02:46.940207 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:46.940177    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:02:46.940541 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:46.940527    2559 scope.go:117] "RemoveContainer" containerID="215b6c162e7b8fb17f5c4879ecdd34a8d573621d2f1c210f0119af0d71ad3360"
Jul 17 19:02:46.940698 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:02:46.940679    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:02:46.975232 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:46.975201    2559 generic.go:356] "Generic (PLEG): container finished" podID="5a359e9b-33ff-4bbe-87ab-26e8320533d8" containerID="4cdfb4e5eebe9c4cb282d81c00a0ee637383083c0bfe228ac9516a197b5bee48" exitCode=7
Jul 17 19:02:46.975232 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:46.975234    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j" event={"ID":"5a359e9b-33ff-4bbe-87ab-26e8320533d8","Type":"ContainerDied","Data":"4cdfb4e5eebe9c4cb282d81c00a0ee637383083c0bfe228ac9516a197b5bee48"}
Jul 17 19:02:46.975749 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:46.975479    2559 scope.go:117] "RemoveContainer" containerID="4cdfb4e5eebe9c4cb282d81c00a0ee637383083c0bfe228ac9516a197b5bee48"
Jul 17 19:02:47.983324 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:47.983274    2559 generic.go:356] "Generic (PLEG): container finished" podID="5a359e9b-33ff-4bbe-87ab-26e8320533d8" containerID="db5b6342e0371d1688e1ebcdfc5acf2e2f29eb20370ed33e28faa5b3e74152e8" exitCode=7
Jul 17 19:02:47.983757 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:47.983338    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j" event={"ID":"5a359e9b-33ff-4bbe-87ab-26e8320533d8","Type":"ContainerDied","Data":"db5b6342e0371d1688e1ebcdfc5acf2e2f29eb20370ed33e28faa5b3e74152e8"}
Jul 17 19:02:47.983757 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:47.983407    2559 scope.go:117] "RemoveContainer" containerID="4cdfb4e5eebe9c4cb282d81c00a0ee637383083c0bfe228ac9516a197b5bee48"
Jul 17 19:02:47.983757 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:47.983588    2559 scope.go:117] "RemoveContainer" containerID="db5b6342e0371d1688e1ebcdfc5acf2e2f29eb20370ed33e28faa5b3e74152e8"
Jul 17 19:02:47.983869 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:02:47.983789    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j_odh-ai-gateway-infra(5a359e9b-33ff-4bbe-87ab-26e8320533d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j" podUID="5a359e9b-33ff-4bbe-87ab-26e8320533d8"
Jul 17 19:02:48.990475 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:48.990393    2559 scope.go:117] "RemoveContainer" containerID="db5b6342e0371d1688e1ebcdfc5acf2e2f29eb20370ed33e28faa5b3e74152e8"
Jul 17 19:02:48.990815 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:02:48.990579    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j_odh-ai-gateway-infra(5a359e9b-33ff-4bbe-87ab-26e8320533d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j" podUID="5a359e9b-33ff-4bbe-87ab-26e8320533d8"
Jul 17 19:02:50.988265 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:50.988234    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:02:50.988681 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:02:50.988614    2559 scope.go:117] "RemoveContainer" containerID="215b6c162e7b8fb17f5c4879ecdd34a8d573621d2f1c210f0119af0d71ad3360"
Jul 17 19:02:50.988798 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:02:50.988780    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:03:02.120266 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:02.120179    2559 scope.go:117] "RemoveContainer" containerID="db5b6342e0371d1688e1ebcdfc5acf2e2f29eb20370ed33e28faa5b3e74152e8"
Jul 17 19:03:03.082071 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:03.082037    2559 generic.go:356] "Generic (PLEG): container finished" podID="5a359e9b-33ff-4bbe-87ab-26e8320533d8" containerID="60858890be756a6217221c7831787a44deff1ac098579f358b5ad12655f0a64d" exitCode=7
Jul 17 19:03:03.082249 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:03.082113    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j" event={"ID":"5a359e9b-33ff-4bbe-87ab-26e8320533d8","Type":"ContainerDied","Data":"60858890be756a6217221c7831787a44deff1ac098579f358b5ad12655f0a64d"}
Jul 17 19:03:03.082249 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:03.082188    2559 scope.go:117] "RemoveContainer" containerID="db5b6342e0371d1688e1ebcdfc5acf2e2f29eb20370ed33e28faa5b3e74152e8"
Jul 17 19:03:03.082538 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:03.082518    2559 scope.go:117] "RemoveContainer" containerID="60858890be756a6217221c7831787a44deff1ac098579f358b5ad12655f0a64d"
Jul 17 19:03:03.082748 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:03:03.082725    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j_odh-ai-gateway-infra(5a359e9b-33ff-4bbe-87ab-26e8320533d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j" podUID="5a359e9b-33ff-4bbe-87ab-26e8320533d8"
Jul 17 19:03:04.113099 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:04.113066    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j"]
Jul 17 19:03:04.238350 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:04.238296    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j"
Jul 17 19:03:04.367360 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:04.367242    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a359e9b-33ff-4bbe-87ab-26e8320533d8-maas-api-service-ca\") pod \"5a359e9b-33ff-4bbe-87ab-26e8320533d8\" (UID: \"5a359e9b-33ff-4bbe-87ab-26e8320533d8\") "
Jul 17 19:03:04.367602 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:04.367577    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a359e9b-33ff-4bbe-87ab-26e8320533d8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5a359e9b-33ff-4bbe-87ab-26e8320533d8" (UID: "5a359e9b-33ff-4bbe-87ab-26e8320533d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:03:04.468894 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:04.468860    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a359e9b-33ff-4bbe-87ab-26e8320533d8-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:03:05.096739 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:05.096715    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j"
Jul 17 19:03:05.096931 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:05.096746    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j" event={"ID":"5a359e9b-33ff-4bbe-87ab-26e8320533d8","Type":"ContainerDied","Data":"4417b55ba09c8acef59da9690cead6a8eb9722f7ff40f87f102c796e9fa4f809"}
Jul 17 19:03:05.096931 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:05.096806    2559 scope.go:117] "RemoveContainer" containerID="60858890be756a6217221c7831787a44deff1ac098579f358b5ad12655f0a64d"
Jul 17 19:03:05.118215 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:05.118189    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j"]
Jul 17 19:03:05.124589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:05.124564    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-fmp9j"]
Jul 17 19:03:06.120515 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:06.120483    2559 scope.go:117] "RemoveContainer" containerID="215b6c162e7b8fb17f5c4879ecdd34a8d573621d2f1c210f0119af0d71ad3360"
Jul 17 19:03:06.126764 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:06.126741    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a359e9b-33ff-4bbe-87ab-26e8320533d8" path="/var/lib/kubelet/pods/5a359e9b-33ff-4bbe-87ab-26e8320533d8/volumes"
Jul 17 19:03:07.111397 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:07.111365    2559 generic.go:356] "Generic (PLEG): container finished" podID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerID="a4383472e153c9f63fcf071af1b82dfbda07e036dde97024a10dca8f03f53e82" exitCode=1
Jul 17 19:03:07.111593 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:07.111436    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" event={"ID":"abe06fcb-35bf-44df-b010-c65504e8a47f","Type":"ContainerDied","Data":"a4383472e153c9f63fcf071af1b82dfbda07e036dde97024a10dca8f03f53e82"}
Jul 17 19:03:07.111593 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:07.111495    2559 scope.go:117] "RemoveContainer" containerID="215b6c162e7b8fb17f5c4879ecdd34a8d573621d2f1c210f0119af0d71ad3360"
Jul 17 19:03:07.111820 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:07.111802    2559 scope.go:117] "RemoveContainer" containerID="a4383472e153c9f63fcf071af1b82dfbda07e036dde97024a10dca8f03f53e82"
Jul 17 19:03:07.112014 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:03:07.111992    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:03:10.226154 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:10.226131    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 19:03:10.231949 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:10.231925    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 19:03:10.234037 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:10.234019    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 19:03:10.239398 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:10.239376    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 19:03:10.988380 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:10.988350    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:03:10.988736 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:10.988723    2559 scope.go:117] "RemoveContainer" containerID="a4383472e153c9f63fcf071af1b82dfbda07e036dde97024a10dca8f03f53e82"
Jul 17 19:03:10.988930 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:03:10.988911    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:03:11.940255 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:11.940222    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:03:11.940689 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:11.940640    2559 scope.go:117] "RemoveContainer" containerID="a4383472e153c9f63fcf071af1b82dfbda07e036dde97024a10dca8f03f53e82"
Jul 17 19:03:11.940828 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:03:11.940809    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:03:23.119884 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:23.119851    2559 scope.go:117] "RemoveContainer" containerID="a4383472e153c9f63fcf071af1b82dfbda07e036dde97024a10dca8f03f53e82"
Jul 17 19:03:23.120284 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:03:23.120038    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:03:34.120063 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:34.120028    2559 scope.go:117] "RemoveContainer" containerID="a4383472e153c9f63fcf071af1b82dfbda07e036dde97024a10dca8f03f53e82"
Jul 17 19:03:35.273439 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:35.273358    2559 generic.go:356] "Generic (PLEG): container finished" podID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerID="81db7a3463708ad1eb68b2a7d2ef7e1e96e9872ce99a4da4ce64a4e01dbbafd1" exitCode=1
Jul 17 19:03:35.273439 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:35.273432    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" event={"ID":"abe06fcb-35bf-44df-b010-c65504e8a47f","Type":"ContainerDied","Data":"81db7a3463708ad1eb68b2a7d2ef7e1e96e9872ce99a4da4ce64a4e01dbbafd1"}
Jul 17 19:03:35.273906 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:35.273490    2559 scope.go:117] "RemoveContainer" containerID="a4383472e153c9f63fcf071af1b82dfbda07e036dde97024a10dca8f03f53e82"
Jul 17 19:03:35.273906 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:35.273792    2559 scope.go:117] "RemoveContainer" containerID="81db7a3463708ad1eb68b2a7d2ef7e1e96e9872ce99a4da4ce64a4e01dbbafd1"
Jul 17 19:03:35.274005 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:03:35.273963    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:03:36.940184 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:36.940157    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:03:36.940569 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:36.940467    2559 scope.go:117] "RemoveContainer" containerID="81db7a3463708ad1eb68b2a7d2ef7e1e96e9872ce99a4da4ce64a4e01dbbafd1"
Jul 17 19:03:36.940630 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:03:36.940611    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:03:40.988698 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:40.988662    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:03:40.989294 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:40.989173    2559 scope.go:117] "RemoveContainer" containerID="81db7a3463708ad1eb68b2a7d2ef7e1e96e9872ce99a4da4ce64a4e01dbbafd1"
Jul 17 19:03:40.989447 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:03:40.989422    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:03:53.119598 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:03:53.119567    2559 scope.go:117] "RemoveContainer" containerID="81db7a3463708ad1eb68b2a7d2ef7e1e96e9872ce99a4da4ce64a4e01dbbafd1"
Jul 17 19:03:53.119987 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:03:53.119733    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:04:08.131227 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:08.131200    2559 scope.go:117] "RemoveContainer" containerID="81db7a3463708ad1eb68b2a7d2ef7e1e96e9872ce99a4da4ce64a4e01dbbafd1"
Jul 17 19:04:08.131622 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:04:08.131402    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:04:23.120267 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:23.120233    2559 scope.go:117] "RemoveContainer" containerID="81db7a3463708ad1eb68b2a7d2ef7e1e96e9872ce99a4da4ce64a4e01dbbafd1"
Jul 17 19:04:24.549918 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:24.549887    2559 generic.go:356] "Generic (PLEG): container finished" podID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerID="691ac182a830f810a56a54a0d5bfc0163d41377aa5267c98c673dad3926f2138" exitCode=1
Jul 17 19:04:24.550391 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:24.549966    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" event={"ID":"abe06fcb-35bf-44df-b010-c65504e8a47f","Type":"ContainerDied","Data":"691ac182a830f810a56a54a0d5bfc0163d41377aa5267c98c673dad3926f2138"}
Jul 17 19:04:24.550391 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:24.550038    2559 scope.go:117] "RemoveContainer" containerID="81db7a3463708ad1eb68b2a7d2ef7e1e96e9872ce99a4da4ce64a4e01dbbafd1"
Jul 17 19:04:24.550518 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:24.550411    2559 scope.go:117] "RemoveContainer" containerID="691ac182a830f810a56a54a0d5bfc0163d41377aa5267c98c673dad3926f2138"
Jul 17 19:04:24.550669 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:04:24.550644    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:04:26.940185 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:26.940153    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:04:26.940595 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:26.940517    2559 scope.go:117] "RemoveContainer" containerID="691ac182a830f810a56a54a0d5bfc0163d41377aa5267c98c673dad3926f2138"
Jul 17 19:04:26.940734 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:04:26.940712    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:04:30.989053 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:30.988971    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:04:30.989481 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:30.989348    2559 scope.go:117] "RemoveContainer" containerID="691ac182a830f810a56a54a0d5bfc0163d41377aa5267c98c673dad3926f2138"
Jul 17 19:04:30.989540 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:04:30.989512    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:04:42.120118 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:42.120083    2559 scope.go:117] "RemoveContainer" containerID="691ac182a830f810a56a54a0d5bfc0163d41377aa5267c98c673dad3926f2138"
Jul 17 19:04:42.120519 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:04:42.120278    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:04:45.025230 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:45.025193    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb"]
Jul 17 19:04:45.025692 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:45.025655    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a359e9b-33ff-4bbe-87ab-26e8320533d8" containerName="revoke-keys"
Jul 17 19:04:45.025692 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:45.025670    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a359e9b-33ff-4bbe-87ab-26e8320533d8" containerName="revoke-keys"
Jul 17 19:04:45.025692 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:45.025689    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a359e9b-33ff-4bbe-87ab-26e8320533d8" containerName="revoke-keys"
Jul 17 19:04:45.025858 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:45.025698    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a359e9b-33ff-4bbe-87ab-26e8320533d8" containerName="revoke-keys"
Jul 17 19:04:45.025858 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:45.025711    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a359e9b-33ff-4bbe-87ab-26e8320533d8" containerName="revoke-keys"
Jul 17 19:04:45.025858 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:45.025716    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a359e9b-33ff-4bbe-87ab-26e8320533d8" containerName="revoke-keys"
Jul 17 19:04:45.025858 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:45.025787    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a359e9b-33ff-4bbe-87ab-26e8320533d8" containerName="revoke-keys"
Jul 17 19:04:45.025858 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:45.025800    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a359e9b-33ff-4bbe-87ab-26e8320533d8" containerName="revoke-keys"
Jul 17 19:04:45.028773 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:45.028755    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb"
Jul 17 19:04:45.031479 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:45.031455    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:04:45.036205 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:45.036186    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb"]
Jul 17 19:04:45.127002 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:45.126972    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a698813-eae1-41ff-a3b3-c454a1ec9a9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb\" (UID: \"8a698813-eae1-41ff-a3b3-c454a1ec9a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb"
Jul 17 19:04:45.228060 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:45.228031    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a698813-eae1-41ff-a3b3-c454a1ec9a9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb\" (UID: \"8a698813-eae1-41ff-a3b3-c454a1ec9a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb"
Jul 17 19:04:45.228768 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:45.228744    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a698813-eae1-41ff-a3b3-c454a1ec9a9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb\" (UID: \"8a698813-eae1-41ff-a3b3-c454a1ec9a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb"
Jul 17 19:04:45.340912 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:45.340881    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb"
Jul 17 19:04:45.472643 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:45.472623    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb"]
Jul 17 19:04:45.474900 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:04:45.474872    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a698813_eae1_41ff_a3b3_c454a1ec9a9d.slice/crio-6c574a5ee6cc86a5c6f7c9c4b43b0741ba85ca73198372c8e945d0b6dcdfaa67 WatchSource:0}: Error finding container 6c574a5ee6cc86a5c6f7c9c4b43b0741ba85ca73198372c8e945d0b6dcdfaa67: Status 404 returned error can't find the container with id 6c574a5ee6cc86a5c6f7c9c4b43b0741ba85ca73198372c8e945d0b6dcdfaa67
Jul 17 19:04:45.676735 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:45.676707    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb" event={"ID":"8a698813-eae1-41ff-a3b3-c454a1ec9a9d","Type":"ContainerStarted","Data":"6c574a5ee6cc86a5c6f7c9c4b43b0741ba85ca73198372c8e945d0b6dcdfaa67"}
Jul 17 19:04:46.683810 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:46.683779    2559 generic.go:356] "Generic (PLEG): container finished" podID="8a698813-eae1-41ff-a3b3-c454a1ec9a9d" containerID="562020eb1f6f31950564f5c34a6dfe6a460a4af2c53b33ac861b4186f4d498bb" exitCode=7
Jul 17 19:04:46.684198 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:46.683874    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb" event={"ID":"8a698813-eae1-41ff-a3b3-c454a1ec9a9d","Type":"ContainerDied","Data":"562020eb1f6f31950564f5c34a6dfe6a460a4af2c53b33ac861b4186f4d498bb"}
Jul 17 19:04:46.684198 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:46.683990    2559 scope.go:117] "RemoveContainer" containerID="562020eb1f6f31950564f5c34a6dfe6a460a4af2c53b33ac861b4186f4d498bb"
Jul 17 19:04:47.691470 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:47.691435    2559 generic.go:356] "Generic (PLEG): container finished" podID="8a698813-eae1-41ff-a3b3-c454a1ec9a9d" containerID="3f01324f05add4f0865a7d43676a585458ad89acfdf803d353863c73aebed8d3" exitCode=7
Jul 17 19:04:47.691871 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:47.691515    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb" event={"ID":"8a698813-eae1-41ff-a3b3-c454a1ec9a9d","Type":"ContainerDied","Data":"3f01324f05add4f0865a7d43676a585458ad89acfdf803d353863c73aebed8d3"}
Jul 17 19:04:47.691871 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:47.691600    2559 scope.go:117] "RemoveContainer" containerID="562020eb1f6f31950564f5c34a6dfe6a460a4af2c53b33ac861b4186f4d498bb"
Jul 17 19:04:47.691871 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:47.691713    2559 scope.go:117] "RemoveContainer" containerID="3f01324f05add4f0865a7d43676a585458ad89acfdf803d353863c73aebed8d3"
Jul 17 19:04:47.692045 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:04:47.691894    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb_odh-ai-gateway-infra(8a698813-eae1-41ff-a3b3-c454a1ec9a9d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb" podUID="8a698813-eae1-41ff-a3b3-c454a1ec9a9d"
Jul 17 19:04:48.699681 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:48.699652    2559 scope.go:117] "RemoveContainer" containerID="3f01324f05add4f0865a7d43676a585458ad89acfdf803d353863c73aebed8d3"
Jul 17 19:04:48.700072 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:04:48.699820    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb_odh-ai-gateway-infra(8a698813-eae1-41ff-a3b3-c454a1ec9a9d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb" podUID="8a698813-eae1-41ff-a3b3-c454a1ec9a9d"
Jul 17 19:04:56.126509 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:04:56.126482    2559 scope.go:117] "RemoveContainer" containerID="691ac182a830f810a56a54a0d5bfc0163d41377aa5267c98c673dad3926f2138"
Jul 17 19:04:56.126964 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:04:56.126649    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:05:02.119643 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:02.119617    2559 scope.go:117] "RemoveContainer" containerID="3f01324f05add4f0865a7d43676a585458ad89acfdf803d353863c73aebed8d3"
Jul 17 19:05:02.786815 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:02.786789    2559 generic.go:356] "Generic (PLEG): container finished" podID="8a698813-eae1-41ff-a3b3-c454a1ec9a9d" containerID="eb0d6d7e44e9b1d703771e140983a266a017804165ead3da07aa00d32dd9bb8a" exitCode=7
Jul 17 19:05:02.786927 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:02.786870    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb" event={"ID":"8a698813-eae1-41ff-a3b3-c454a1ec9a9d","Type":"ContainerDied","Data":"eb0d6d7e44e9b1d703771e140983a266a017804165ead3da07aa00d32dd9bb8a"}
Jul 17 19:05:02.786969 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:02.786932    2559 scope.go:117] "RemoveContainer" containerID="3f01324f05add4f0865a7d43676a585458ad89acfdf803d353863c73aebed8d3"
Jul 17 19:05:02.787174 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:02.787151    2559 scope.go:117] "RemoveContainer" containerID="eb0d6d7e44e9b1d703771e140983a266a017804165ead3da07aa00d32dd9bb8a"
Jul 17 19:05:02.787398 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:05:02.787379    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb_odh-ai-gateway-infra(8a698813-eae1-41ff-a3b3-c454a1ec9a9d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb" podUID="8a698813-eae1-41ff-a3b3-c454a1ec9a9d"
Jul 17 19:05:03.812753 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:03.812723    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb"]
Jul 17 19:05:03.946018 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:03.945992    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb"
Jul 17 19:05:04.084655 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:04.084575    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a698813-eae1-41ff-a3b3-c454a1ec9a9d-maas-api-service-ca\") pod \"8a698813-eae1-41ff-a3b3-c454a1ec9a9d\" (UID: \"8a698813-eae1-41ff-a3b3-c454a1ec9a9d\") "
Jul 17 19:05:04.084965 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:04.084935    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a698813-eae1-41ff-a3b3-c454a1ec9a9d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8a698813-eae1-41ff-a3b3-c454a1ec9a9d" (UID: "8a698813-eae1-41ff-a3b3-c454a1ec9a9d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:05:04.185397 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:04.185366    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a698813-eae1-41ff-a3b3-c454a1ec9a9d-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:05:04.801500 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:04.801471    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb"
Jul 17 19:05:04.801674 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:04.801504    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb" event={"ID":"8a698813-eae1-41ff-a3b3-c454a1ec9a9d","Type":"ContainerDied","Data":"6c574a5ee6cc86a5c6f7c9c4b43b0741ba85ca73198372c8e945d0b6dcdfaa67"}
Jul 17 19:05:04.801674 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:04.801566    2559 scope.go:117] "RemoveContainer" containerID="eb0d6d7e44e9b1d703771e140983a266a017804165ead3da07aa00d32dd9bb8a"
Jul 17 19:05:04.818213 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:04.818188    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb"]
Jul 17 19:05:04.821151 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:04.821120    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-26ctb"]
Jul 17 19:05:06.125917 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:06.125884    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a698813-eae1-41ff-a3b3-c454a1ec9a9d" path="/var/lib/kubelet/pods/8a698813-eae1-41ff-a3b3-c454a1ec9a9d/volumes"
Jul 17 19:05:09.119749 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:09.119724    2559 scope.go:117] "RemoveContainer" containerID="691ac182a830f810a56a54a0d5bfc0163d41377aa5267c98c673dad3926f2138"
Jul 17 19:05:09.120119 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:05:09.119884    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:05:24.119734 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:24.119702    2559 scope.go:117] "RemoveContainer" containerID="691ac182a830f810a56a54a0d5bfc0163d41377aa5267c98c673dad3926f2138"
Jul 17 19:05:24.122104 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:05:24.119938    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:05:35.119451 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:35.119422    2559 scope.go:117] "RemoveContainer" containerID="691ac182a830f810a56a54a0d5bfc0163d41377aa5267c98c673dad3926f2138"
Jul 17 19:05:35.120028 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:05:35.119603    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:05:47.119434 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:47.119405    2559 scope.go:117] "RemoveContainer" containerID="691ac182a830f810a56a54a0d5bfc0163d41377aa5267c98c673dad3926f2138"
Jul 17 19:05:47.119867 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:47.119702    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 19:05:48.046649 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:48.046573    2559 generic.go:356] "Generic (PLEG): container finished" podID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerID="9632690ff8959230287d1db42aaf61811229ac27c41a199c97498e3fe3ee358d" exitCode=1
Jul 17 19:05:48.046779 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:48.046652    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" event={"ID":"abe06fcb-35bf-44df-b010-c65504e8a47f","Type":"ContainerDied","Data":"9632690ff8959230287d1db42aaf61811229ac27c41a199c97498e3fe3ee358d"}
Jul 17 19:05:48.046779 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:48.046742    2559 scope.go:117] "RemoveContainer" containerID="691ac182a830f810a56a54a0d5bfc0163d41377aa5267c98c673dad3926f2138"
Jul 17 19:05:48.047067 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:48.047050    2559 scope.go:117] "RemoveContainer" containerID="9632690ff8959230287d1db42aaf61811229ac27c41a199c97498e3fe3ee358d"
Jul 17 19:05:48.047266 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:05:48.047247    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c_odh-ai-gateway-infra(abe06fcb-35bf-44df-b010-c65504e8a47f)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f"
Jul 17 19:05:50.923943 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:50.923911    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"]
Jul 17 19:05:51.053218 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:51.053192    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:05:51.065702 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:51.065685    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"
Jul 17 19:05:51.065820 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:51.065712    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c" event={"ID":"abe06fcb-35bf-44df-b010-c65504e8a47f","Type":"ContainerDied","Data":"1a1b0f51f0cdbb21f7ede65f866fe4ee7142bd40f40417bde9c2428654dd135a"}
Jul 17 19:05:51.065820 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:51.065764    2559 scope.go:117] "RemoveContainer" containerID="9632690ff8959230287d1db42aaf61811229ac27c41a199c97498e3fe3ee358d"
Jul 17 19:05:51.164612 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:51.164589    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqnnr\" (UniqueName: \"kubernetes.io/projected/abe06fcb-35bf-44df-b010-c65504e8a47f-kube-api-access-lqnnr\") pod \"abe06fcb-35bf-44df-b010-c65504e8a47f\" (UID: \"abe06fcb-35bf-44df-b010-c65504e8a47f\") "
Jul 17 19:05:51.164732 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:51.164701    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe06fcb-35bf-44df-b010-c65504e8a47f-maas-api-tls\") pod \"abe06fcb-35bf-44df-b010-c65504e8a47f\" (UID: \"abe06fcb-35bf-44df-b010-c65504e8a47f\") "
Jul 17 19:05:51.164732 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:51.164730    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe06fcb-35bf-44df-b010-c65504e8a47f-openshift-service-ca\") pod \"abe06fcb-35bf-44df-b010-c65504e8a47f\" (UID: \"abe06fcb-35bf-44df-b010-c65504e8a47f\") "
Jul 17 19:05:51.165160 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:51.165093    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe06fcb-35bf-44df-b010-c65504e8a47f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "abe06fcb-35bf-44df-b010-c65504e8a47f" (UID: "abe06fcb-35bf-44df-b010-c65504e8a47f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:05:51.166877 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:51.166858    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe06fcb-35bf-44df-b010-c65504e8a47f-kube-api-access-lqnnr" (OuterVolumeSpecName: "kube-api-access-lqnnr") pod "abe06fcb-35bf-44df-b010-c65504e8a47f" (UID: "abe06fcb-35bf-44df-b010-c65504e8a47f"). InnerVolumeSpecName "kube-api-access-lqnnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:05:51.166968 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:51.166888    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe06fcb-35bf-44df-b010-c65504e8a47f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "abe06fcb-35bf-44df-b010-c65504e8a47f" (UID: "abe06fcb-35bf-44df-b010-c65504e8a47f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:05:51.265544 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:51.265483    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe06fcb-35bf-44df-b010-c65504e8a47f-maas-api-tls\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:05:51.265544 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:51.265508    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe06fcb-35bf-44df-b010-c65504e8a47f-openshift-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:05:51.265544 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:51.265523    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqnnr\" (UniqueName: \"kubernetes.io/projected/abe06fcb-35bf-44df-b010-c65504e8a47f-kube-api-access-lqnnr\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:05:51.387913 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:51.387890    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"]
Jul 17 19:05:51.391759 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:51.391737    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7dc4fd53-659d6b7c87-n829c"]
Jul 17 19:05:52.131131 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:05:52.131089    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f" path="/var/lib/kubelet/pods/abe06fcb-35bf-44df-b010-c65504e8a47f/volumes"
Jul 17 19:06:03.282190 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282106    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr"]
Jul 17 19:06:03.282810 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282680    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerName="maas-api"
Jul 17 19:06:03.282810 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282698    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerName="maas-api"
Jul 17 19:06:03.282810 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282715    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerName="maas-api"
Jul 17 19:06:03.282810 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282723    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerName="maas-api"
Jul 17 19:06:03.282810 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282735    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerName="maas-api"
Jul 17 19:06:03.282810 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282743    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerName="maas-api"
Jul 17 19:06:03.282810 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282764    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerName="maas-api"
Jul 17 19:06:03.282810 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282772    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerName="maas-api"
Jul 17 19:06:03.282810 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282784    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerName="maas-api"
Jul 17 19:06:03.282810 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282792    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerName="maas-api"
Jul 17 19:06:03.282810 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282803    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a698813-eae1-41ff-a3b3-c454a1ec9a9d" containerName="revoke-keys"
Jul 17 19:06:03.282810 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282812    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a698813-eae1-41ff-a3b3-c454a1ec9a9d" containerName="revoke-keys"
Jul 17 19:06:03.283189 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282835    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a698813-eae1-41ff-a3b3-c454a1ec9a9d" containerName="revoke-keys"
Jul 17 19:06:03.283189 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282843    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a698813-eae1-41ff-a3b3-c454a1ec9a9d" containerName="revoke-keys"
Jul 17 19:06:03.283189 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282926    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerName="maas-api"
Jul 17 19:06:03.283189 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282936    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerName="maas-api"
Jul 17 19:06:03.283189 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282947    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a698813-eae1-41ff-a3b3-c454a1ec9a9d" containerName="revoke-keys"
Jul 17 19:06:03.283189 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282956    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerName="maas-api"
Jul 17 19:06:03.283189 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282965    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerName="maas-api"
Jul 17 19:06:03.283189 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282975    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a359e9b-33ff-4bbe-87ab-26e8320533d8" containerName="revoke-keys"
Jul 17 19:06:03.283189 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.282983    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a698813-eae1-41ff-a3b3-c454a1ec9a9d" containerName="revoke-keys"
Jul 17 19:06:03.287707 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.287690    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr"
Jul 17 19:06:03.290534 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.290509    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:06:03.292864 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.292833    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr"]
Jul 17 19:06:03.465696 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.465666    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/839aa379-f251-4cb8-a9eb-924352f39763-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr\" (UID: \"839aa379-f251-4cb8-a9eb-924352f39763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr"
Jul 17 19:06:03.567030 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.566995    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/839aa379-f251-4cb8-a9eb-924352f39763-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr\" (UID: \"839aa379-f251-4cb8-a9eb-924352f39763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr"
Jul 17 19:06:03.567634 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.567616    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/839aa379-f251-4cb8-a9eb-924352f39763-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr\" (UID: \"839aa379-f251-4cb8-a9eb-924352f39763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr"
Jul 17 19:06:03.600739 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.600705    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr"
Jul 17 19:06:03.744268 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:03.744231    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr"]
Jul 17 19:06:03.745165 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:06:03.745128    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839aa379_f251_4cb8_a9eb_924352f39763.slice/crio-7a01d29793e0788630da1bacba7ed397357c772bc1d71a0126dd8ebba42072e8 WatchSource:0}: Error finding container 7a01d29793e0788630da1bacba7ed397357c772bc1d71a0126dd8ebba42072e8: Status 404 returned error can't find the container with id 7a01d29793e0788630da1bacba7ed397357c772bc1d71a0126dd8ebba42072e8
Jul 17 19:06:04.143757 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:04.143716    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr" event={"ID":"839aa379-f251-4cb8-a9eb-924352f39763","Type":"ContainerStarted","Data":"93ef9c513cf06957fdc9333a32353432083d7d9deedb0d92a93933cc58117a45"}
Jul 17 19:06:04.143970 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:04.143778    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr" event={"ID":"839aa379-f251-4cb8-a9eb-924352f39763","Type":"ContainerStarted","Data":"7a01d29793e0788630da1bacba7ed397357c772bc1d71a0126dd8ebba42072e8"}
Jul 17 19:06:04.158850 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:04.158803    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr" podStartSLOduration=1.158786718 podStartE2EDuration="1.158786718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:06:04.157540786 +0000 UTC m=+1974.599049475" watchObservedRunningTime="2026-07-17 19:06:04.158786718 +0000 UTC m=+1974.600295410"
Jul 17 19:06:05.151034 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:05.151003    2559 generic.go:356] "Generic (PLEG): container finished" podID="839aa379-f251-4cb8-a9eb-924352f39763" containerID="93ef9c513cf06957fdc9333a32353432083d7d9deedb0d92a93933cc58117a45" exitCode=7
Jul 17 19:06:05.151493 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:05.151087    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr" event={"ID":"839aa379-f251-4cb8-a9eb-924352f39763","Type":"ContainerDied","Data":"93ef9c513cf06957fdc9333a32353432083d7d9deedb0d92a93933cc58117a45"}
Jul 17 19:06:05.151493 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:05.151220    2559 scope.go:117] "RemoveContainer" containerID="93ef9c513cf06957fdc9333a32353432083d7d9deedb0d92a93933cc58117a45"
Jul 17 19:06:06.158155 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:06.158121    2559 generic.go:356] "Generic (PLEG): container finished" podID="839aa379-f251-4cb8-a9eb-924352f39763" containerID="cc10fd42b0a50dfe0e1122f3eab6a542921ef7c5acb7b4ef4c1fa0d95a356213" exitCode=7
Jul 17 19:06:06.158655 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:06.158220    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr" event={"ID":"839aa379-f251-4cb8-a9eb-924352f39763","Type":"ContainerDied","Data":"cc10fd42b0a50dfe0e1122f3eab6a542921ef7c5acb7b4ef4c1fa0d95a356213"}
Jul 17 19:06:06.158655 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:06.158285    2559 scope.go:117] "RemoveContainer" containerID="93ef9c513cf06957fdc9333a32353432083d7d9deedb0d92a93933cc58117a45"
Jul 17 19:06:06.158655 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:06.158404    2559 scope.go:117] "RemoveContainer" containerID="cc10fd42b0a50dfe0e1122f3eab6a542921ef7c5acb7b4ef4c1fa0d95a356213"
Jul 17 19:06:06.158655 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:06:06.158622    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr_odh-ai-gateway-infra(839aa379-f251-4cb8-a9eb-924352f39763)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr" podUID="839aa379-f251-4cb8-a9eb-924352f39763"
Jul 17 19:06:07.166189 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:07.166158    2559 scope.go:117] "RemoveContainer" containerID="cc10fd42b0a50dfe0e1122f3eab6a542921ef7c5acb7b4ef4c1fa0d95a356213"
Jul 17 19:06:07.166620 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:06:07.166425    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr_odh-ai-gateway-infra(839aa379-f251-4cb8-a9eb-924352f39763)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr" podUID="839aa379-f251-4cb8-a9eb-924352f39763"
Jul 17 19:06:21.120134 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:21.120108    2559 scope.go:117] "RemoveContainer" containerID="cc10fd42b0a50dfe0e1122f3eab6a542921ef7c5acb7b4ef4c1fa0d95a356213"
Jul 17 19:06:22.247057 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:22.247023    2559 generic.go:356] "Generic (PLEG): container finished" podID="839aa379-f251-4cb8-a9eb-924352f39763" containerID="993fa28c07f3a88dd57710df274a2bd4d83aea4885680ef74402e37ffaf19325" exitCode=7
Jul 17 19:06:22.247477 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:22.247091    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr" event={"ID":"839aa379-f251-4cb8-a9eb-924352f39763","Type":"ContainerDied","Data":"993fa28c07f3a88dd57710df274a2bd4d83aea4885680ef74402e37ffaf19325"}
Jul 17 19:06:22.247477 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:22.247151    2559 scope.go:117] "RemoveContainer" containerID="cc10fd42b0a50dfe0e1122f3eab6a542921ef7c5acb7b4ef4c1fa0d95a356213"
Jul 17 19:06:22.247477 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:22.247448    2559 scope.go:117] "RemoveContainer" containerID="993fa28c07f3a88dd57710df274a2bd4d83aea4885680ef74402e37ffaf19325"
Jul 17 19:06:22.247667 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:06:22.247648    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr_odh-ai-gateway-infra(839aa379-f251-4cb8-a9eb-924352f39763)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr" podUID="839aa379-f251-4cb8-a9eb-924352f39763"
Jul 17 19:06:23.285647 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:23.285614    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr"]
Jul 17 19:06:23.421560 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:23.421538    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr"
Jul 17 19:06:23.548185 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:23.548097    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/839aa379-f251-4cb8-a9eb-924352f39763-maas-api-service-ca\") pod \"839aa379-f251-4cb8-a9eb-924352f39763\" (UID: \"839aa379-f251-4cb8-a9eb-924352f39763\") "
Jul 17 19:06:23.548493 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:23.548471    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839aa379-f251-4cb8-a9eb-924352f39763-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "839aa379-f251-4cb8-a9eb-924352f39763" (UID: "839aa379-f251-4cb8-a9eb-924352f39763"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:06:23.649245 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:23.649211    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/839aa379-f251-4cb8-a9eb-924352f39763-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:06:24.261763 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:24.261723    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr" event={"ID":"839aa379-f251-4cb8-a9eb-924352f39763","Type":"ContainerDied","Data":"7a01d29793e0788630da1bacba7ed397357c772bc1d71a0126dd8ebba42072e8"}
Jul 17 19:06:24.261955 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:24.261785    2559 scope.go:117] "RemoveContainer" containerID="993fa28c07f3a88dd57710df274a2bd4d83aea4885680ef74402e37ffaf19325"
Jul 17 19:06:24.261955 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:24.261742    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr"
Jul 17 19:06:24.278809 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:24.278780    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr"]
Jul 17 19:06:24.282701 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:24.282677    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-txjsr"]
Jul 17 19:06:25.112800 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.112766    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2"]
Jul 17 19:06:25.113200 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.113129    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="839aa379-f251-4cb8-a9eb-924352f39763" containerName="revoke-keys"
Jul 17 19:06:25.113200 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.113140    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="839aa379-f251-4cb8-a9eb-924352f39763" containerName="revoke-keys"
Jul 17 19:06:25.113200 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.113162    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="839aa379-f251-4cb8-a9eb-924352f39763" containerName="revoke-keys"
Jul 17 19:06:25.113200 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.113191    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="839aa379-f251-4cb8-a9eb-924352f39763" containerName="revoke-keys"
Jul 17 19:06:25.113200 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.113200    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerName="maas-api"
Jul 17 19:06:25.113441 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.113207    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerName="maas-api"
Jul 17 19:06:25.113441 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.113215    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a698813-eae1-41ff-a3b3-c454a1ec9a9d" containerName="revoke-keys"
Jul 17 19:06:25.113441 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.113220    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a698813-eae1-41ff-a3b3-c454a1ec9a9d" containerName="revoke-keys"
Jul 17 19:06:25.113441 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.113227    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="839aa379-f251-4cb8-a9eb-924352f39763" containerName="revoke-keys"
Jul 17 19:06:25.113441 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.113232    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="839aa379-f251-4cb8-a9eb-924352f39763" containerName="revoke-keys"
Jul 17 19:06:25.113441 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.113319    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a698813-eae1-41ff-a3b3-c454a1ec9a9d" containerName="revoke-keys"
Jul 17 19:06:25.113441 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.113327    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="839aa379-f251-4cb8-a9eb-924352f39763" containerName="revoke-keys"
Jul 17 19:06:25.113441 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.113332    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerName="maas-api"
Jul 17 19:06:25.113441 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.113339    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="839aa379-f251-4cb8-a9eb-924352f39763" containerName="revoke-keys"
Jul 17 19:06:25.117751 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.117731    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2"
Jul 17 19:06:25.120213 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.120197    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:06:25.124604 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.124582    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2"]
Jul 17 19:06:25.161815 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.161775    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9201cb13-a01a-4e99-86fb-af80547714e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2\" (UID: \"9201cb13-a01a-4e99-86fb-af80547714e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2"
Jul 17 19:06:25.262912 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.262882    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9201cb13-a01a-4e99-86fb-af80547714e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2\" (UID: \"9201cb13-a01a-4e99-86fb-af80547714e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2"
Jul 17 19:06:25.263493 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.263474    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9201cb13-a01a-4e99-86fb-af80547714e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2\" (UID: \"9201cb13-a01a-4e99-86fb-af80547714e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2"
Jul 17 19:06:25.431815 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.431734    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2"
Jul 17 19:06:25.567503 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:25.567473    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2"]
Jul 17 19:06:25.569361 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:06:25.569334    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9201cb13_a01a_4e99_86fb_af80547714e0.slice/crio-1c3b736d540aa28833c44d5c798efd6b7ed588ba367ee48e4cd7570e6d21f324 WatchSource:0}: Error finding container 1c3b736d540aa28833c44d5c798efd6b7ed588ba367ee48e4cd7570e6d21f324: Status 404 returned error can't find the container with id 1c3b736d540aa28833c44d5c798efd6b7ed588ba367ee48e4cd7570e6d21f324
Jul 17 19:06:26.125659 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:26.125630    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="839aa379-f251-4cb8-a9eb-924352f39763" path="/var/lib/kubelet/pods/839aa379-f251-4cb8-a9eb-924352f39763/volumes"
Jul 17 19:06:26.275510 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:26.275474    2559 generic.go:356] "Generic (PLEG): container finished" podID="9201cb13-a01a-4e99-86fb-af80547714e0" containerID="6ef5a740e3497edd967934cba2e3f049ecb4ce62a2ac5697653b068b5652371b" exitCode=6
Jul 17 19:06:26.275673 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:26.275569    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2" event={"ID":"9201cb13-a01a-4e99-86fb-af80547714e0","Type":"ContainerDied","Data":"6ef5a740e3497edd967934cba2e3f049ecb4ce62a2ac5697653b068b5652371b"}
Jul 17 19:06:26.275673 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:26.275644    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2" event={"ID":"9201cb13-a01a-4e99-86fb-af80547714e0","Type":"ContainerStarted","Data":"1c3b736d540aa28833c44d5c798efd6b7ed588ba367ee48e4cd7570e6d21f324"}
Jul 17 19:06:26.275815 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:26.275754    2559 scope.go:117] "RemoveContainer" containerID="6ef5a740e3497edd967934cba2e3f049ecb4ce62a2ac5697653b068b5652371b"
Jul 17 19:06:27.283429 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:27.283334    2559 generic.go:356] "Generic (PLEG): container finished" podID="9201cb13-a01a-4e99-86fb-af80547714e0" containerID="02b4f2d371de8bf4aee4e118eb0397ca46a61745c35be58176adece833439a30" exitCode=6
Jul 17 19:06:27.283429 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:27.283368    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2" event={"ID":"9201cb13-a01a-4e99-86fb-af80547714e0","Type":"ContainerDied","Data":"02b4f2d371de8bf4aee4e118eb0397ca46a61745c35be58176adece833439a30"}
Jul 17 19:06:27.283429 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:27.283413    2559 scope.go:117] "RemoveContainer" containerID="6ef5a740e3497edd967934cba2e3f049ecb4ce62a2ac5697653b068b5652371b"
Jul 17 19:06:27.283988 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:27.283610    2559 scope.go:117] "RemoveContainer" containerID="02b4f2d371de8bf4aee4e118eb0397ca46a61745c35be58176adece833439a30"
Jul 17 19:06:27.283988 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:06:27.283815    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2_odh-ai-gateway-infra(9201cb13-a01a-4e99-86fb-af80547714e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2" podUID="9201cb13-a01a-4e99-86fb-af80547714e0"
Jul 17 19:06:28.290188 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:28.290162    2559 scope.go:117] "RemoveContainer" containerID="02b4f2d371de8bf4aee4e118eb0397ca46a61745c35be58176adece833439a30"
Jul 17 19:06:28.290577 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:06:28.290380    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2_odh-ai-gateway-infra(9201cb13-a01a-4e99-86fb-af80547714e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2" podUID="9201cb13-a01a-4e99-86fb-af80547714e0"
Jul 17 19:06:41.119630 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:41.119604    2559 scope.go:117] "RemoveContainer" containerID="02b4f2d371de8bf4aee4e118eb0397ca46a61745c35be58176adece833439a30"
Jul 17 19:06:42.376661 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:42.376630    2559 generic.go:356] "Generic (PLEG): container finished" podID="9201cb13-a01a-4e99-86fb-af80547714e0" containerID="839d785826f557c557371dd01d13ffbd7da430aba1b60ae207e7b1ca75905232" exitCode=6
Jul 17 19:06:42.377113 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:42.376706    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2" event={"ID":"9201cb13-a01a-4e99-86fb-af80547714e0","Type":"ContainerDied","Data":"839d785826f557c557371dd01d13ffbd7da430aba1b60ae207e7b1ca75905232"}
Jul 17 19:06:42.377113 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:42.376770    2559 scope.go:117] "RemoveContainer" containerID="02b4f2d371de8bf4aee4e118eb0397ca46a61745c35be58176adece833439a30"
Jul 17 19:06:42.377113 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:42.377080    2559 scope.go:117] "RemoveContainer" containerID="839d785826f557c557371dd01d13ffbd7da430aba1b60ae207e7b1ca75905232"
Jul 17 19:06:42.377341 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:06:42.377302    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2_odh-ai-gateway-infra(9201cb13-a01a-4e99-86fb-af80547714e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2" podUID="9201cb13-a01a-4e99-86fb-af80547714e0"
Jul 17 19:06:43.404332 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:43.404290    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2"]
Jul 17 19:06:43.540977 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:43.540955    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2"
Jul 17 19:06:43.712152 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:43.712049    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9201cb13-a01a-4e99-86fb-af80547714e0-maas-api-service-ca\") pod \"9201cb13-a01a-4e99-86fb-af80547714e0\" (UID: \"9201cb13-a01a-4e99-86fb-af80547714e0\") "
Jul 17 19:06:43.712475 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:43.712452    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9201cb13-a01a-4e99-86fb-af80547714e0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9201cb13-a01a-4e99-86fb-af80547714e0" (UID: "9201cb13-a01a-4e99-86fb-af80547714e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:06:43.813695 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:43.813666    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9201cb13-a01a-4e99-86fb-af80547714e0-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:06:44.391161 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:44.391077    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2"
Jul 17 19:06:44.391161 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:44.391114    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2" event={"ID":"9201cb13-a01a-4e99-86fb-af80547714e0","Type":"ContainerDied","Data":"1c3b736d540aa28833c44d5c798efd6b7ed588ba367ee48e4cd7570e6d21f324"}
Jul 17 19:06:44.391425 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:44.391183    2559 scope.go:117] "RemoveContainer" containerID="839d785826f557c557371dd01d13ffbd7da430aba1b60ae207e7b1ca75905232"
Jul 17 19:06:44.408796 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:44.408772    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2"]
Jul 17 19:06:44.412838 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:44.412814    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-4trl2"]
Jul 17 19:06:46.125695 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:06:46.125666    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9201cb13-a01a-4e99-86fb-af80547714e0" path="/var/lib/kubelet/pods/9201cb13-a01a-4e99-86fb-af80547714e0/volumes"
Jul 17 19:07:03.337722 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:03.337684    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl"]
Jul 17 19:07:03.338281 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:03.338263    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9201cb13-a01a-4e99-86fb-af80547714e0" containerName="revoke-keys"
Jul 17 19:07:03.338381 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:03.338286    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9201cb13-a01a-4e99-86fb-af80547714e0" containerName="revoke-keys"
Jul 17 19:07:03.338381 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:03.338327    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9201cb13-a01a-4e99-86fb-af80547714e0" containerName="revoke-keys"
Jul 17 19:07:03.338381 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:03.338336    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9201cb13-a01a-4e99-86fb-af80547714e0" containerName="revoke-keys"
Jul 17 19:07:03.338486 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:03.338458    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9201cb13-a01a-4e99-86fb-af80547714e0" containerName="revoke-keys"
Jul 17 19:07:03.338486 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:03.338471    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="839aa379-f251-4cb8-a9eb-924352f39763" containerName="revoke-keys"
Jul 17 19:07:03.338486 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:03.338480    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9201cb13-a01a-4e99-86fb-af80547714e0" containerName="revoke-keys"
Jul 17 19:07:03.338577 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:03.338491    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe06fcb-35bf-44df-b010-c65504e8a47f" containerName="maas-api"
Jul 17 19:07:03.343403 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:03.343383    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl"
Jul 17 19:07:03.348237 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:03.348214    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:07:03.352020 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:03.351990    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl"]
Jul 17 19:07:03.379234 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:03.379206    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82fa5e61-18a1-4169-8db7-277cac1368c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl\" (UID: \"82fa5e61-18a1-4169-8db7-277cac1368c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl"
Jul 17 19:07:03.480432 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:03.480404    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82fa5e61-18a1-4169-8db7-277cac1368c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl\" (UID: \"82fa5e61-18a1-4169-8db7-277cac1368c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl"
Jul 17 19:07:03.480975 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:03.480955    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82fa5e61-18a1-4169-8db7-277cac1368c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl\" (UID: \"82fa5e61-18a1-4169-8db7-277cac1368c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl"
Jul 17 19:07:03.657535 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:03.657462    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl"
Jul 17 19:07:03.798178 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:03.798153    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl"]
Jul 17 19:07:03.800897 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:07:03.800873    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82fa5e61_18a1_4169_8db7_277cac1368c1.slice/crio-f334fa2afce191ce393dcf8a07dd7e94f514732666ba4ec4da02b1e6542412c4 WatchSource:0}: Error finding container f334fa2afce191ce393dcf8a07dd7e94f514732666ba4ec4da02b1e6542412c4: Status 404 returned error can't find the container with id f334fa2afce191ce393dcf8a07dd7e94f514732666ba4ec4da02b1e6542412c4
Jul 17 19:07:04.500675 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:04.500645    2559 generic.go:356] "Generic (PLEG): container finished" podID="82fa5e61-18a1-4169-8db7-277cac1368c1" containerID="5151343f8935073685fd7676abe6567d150d066c5f27d2844c4a78a3fc9cf897" exitCode=7
Jul 17 19:07:04.501027 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:04.500712    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl" event={"ID":"82fa5e61-18a1-4169-8db7-277cac1368c1","Type":"ContainerDied","Data":"5151343f8935073685fd7676abe6567d150d066c5f27d2844c4a78a3fc9cf897"}
Jul 17 19:07:04.501027 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:04.500749    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl" event={"ID":"82fa5e61-18a1-4169-8db7-277cac1368c1","Type":"ContainerStarted","Data":"f334fa2afce191ce393dcf8a07dd7e94f514732666ba4ec4da02b1e6542412c4"}
Jul 17 19:07:04.501027 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:04.500892    2559 scope.go:117] "RemoveContainer" containerID="5151343f8935073685fd7676abe6567d150d066c5f27d2844c4a78a3fc9cf897"
Jul 17 19:07:05.509287 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:05.509204    2559 generic.go:356] "Generic (PLEG): container finished" podID="82fa5e61-18a1-4169-8db7-277cac1368c1" containerID="bd46188920befcd8c14fca2dae064818b38f52af225c61fe76e6cd9998878cb1" exitCode=7
Jul 17 19:07:05.509287 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:05.509267    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl" event={"ID":"82fa5e61-18a1-4169-8db7-277cac1368c1","Type":"ContainerDied","Data":"bd46188920befcd8c14fca2dae064818b38f52af225c61fe76e6cd9998878cb1"}
Jul 17 19:07:05.509775 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:05.509359    2559 scope.go:117] "RemoveContainer" containerID="5151343f8935073685fd7676abe6567d150d066c5f27d2844c4a78a3fc9cf897"
Jul 17 19:07:05.509775 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:05.509511    2559 scope.go:117] "RemoveContainer" containerID="bd46188920befcd8c14fca2dae064818b38f52af225c61fe76e6cd9998878cb1"
Jul 17 19:07:05.509775 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:07:05.509717    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl_odh-ai-gateway-infra(82fa5e61-18a1-4169-8db7-277cac1368c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl" podUID="82fa5e61-18a1-4169-8db7-277cac1368c1"
Jul 17 19:07:06.516983 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:06.516955    2559 scope.go:117] "RemoveContainer" containerID="bd46188920befcd8c14fca2dae064818b38f52af225c61fe76e6cd9998878cb1"
Jul 17 19:07:06.517405 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:07:06.517145    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl_odh-ai-gateway-infra(82fa5e61-18a1-4169-8db7-277cac1368c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl" podUID="82fa5e61-18a1-4169-8db7-277cac1368c1"
Jul 17 19:07:20.123594 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:20.123565    2559 scope.go:117] "RemoveContainer" containerID="bd46188920befcd8c14fca2dae064818b38f52af225c61fe76e6cd9998878cb1"
Jul 17 19:07:20.605630 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:20.605586    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl" event={"ID":"82fa5e61-18a1-4169-8db7-277cac1368c1","Type":"ContainerStarted","Data":"c015b46925cb834f4c50125049f165434da1238e3e7eb53714bb81492bba1efd"}
Jul 17 19:07:20.635349 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:20.635280    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl" podStartSLOduration=17.63526172 podStartE2EDuration="17.63526172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:07:20.632233471 +0000 UTC m=+2051.073742160" watchObservedRunningTime="2026-07-17 19:07:20.63526172 +0000 UTC m=+2051.076770410"
Jul 17 19:07:21.612583 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:21.612550    2559 generic.go:356] "Generic (PLEG): container finished" podID="82fa5e61-18a1-4169-8db7-277cac1368c1" containerID="c015b46925cb834f4c50125049f165434da1238e3e7eb53714bb81492bba1efd" exitCode=6
Jul 17 19:07:21.613016 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:21.612620    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl" event={"ID":"82fa5e61-18a1-4169-8db7-277cac1368c1","Type":"ContainerDied","Data":"c015b46925cb834f4c50125049f165434da1238e3e7eb53714bb81492bba1efd"}
Jul 17 19:07:21.613016 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:21.612682    2559 scope.go:117] "RemoveContainer" containerID="bd46188920befcd8c14fca2dae064818b38f52af225c61fe76e6cd9998878cb1"
Jul 17 19:07:21.613016 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:21.612960    2559 scope.go:117] "RemoveContainer" containerID="c015b46925cb834f4c50125049f165434da1238e3e7eb53714bb81492bba1efd"
Jul 17 19:07:21.613180 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:07:21.613161    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl_odh-ai-gateway-infra(82fa5e61-18a1-4169-8db7-277cac1368c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl" podUID="82fa5e61-18a1-4169-8db7-277cac1368c1"
Jul 17 19:07:21.639637 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:21.639612    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl"]
Jul 17 19:07:22.208412 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:22.208363    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9"]
Jul 17 19:07:22.208802 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:22.208787    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9201cb13-a01a-4e99-86fb-af80547714e0" containerName="revoke-keys"
Jul 17 19:07:22.208862 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:22.208804    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9201cb13-a01a-4e99-86fb-af80547714e0" containerName="revoke-keys"
Jul 17 19:07:22.208908 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:22.208881    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9201cb13-a01a-4e99-86fb-af80547714e0" containerName="revoke-keys"
Jul 17 19:07:22.213121 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:22.213104    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9"
Jul 17 19:07:22.220097 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:22.219986    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9"]
Jul 17 19:07:22.355425 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:22.355387    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4ea9be-ddca-435e-8c65-fc641214fafa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9\" (UID: \"6f4ea9be-ddca-435e-8c65-fc641214fafa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9"
Jul 17 19:07:22.456578 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:22.456546    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4ea9be-ddca-435e-8c65-fc641214fafa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9\" (UID: \"6f4ea9be-ddca-435e-8c65-fc641214fafa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9"
Jul 17 19:07:22.457170 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:22.457151    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4ea9be-ddca-435e-8c65-fc641214fafa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9\" (UID: \"6f4ea9be-ddca-435e-8c65-fc641214fafa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9"
Jul 17 19:07:22.525831 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:22.525758    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9"
Jul 17 19:07:22.667166 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:22.667133    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9"]
Jul 17 19:07:22.670644 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:07:22.670602    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4ea9be_ddca_435e_8c65_fc641214fafa.slice/crio-573324444d59ee26474b51bf1b1c8ff7f42ab58cc23642c50a1de004cd599af9 WatchSource:0}: Error finding container 573324444d59ee26474b51bf1b1c8ff7f42ab58cc23642c50a1de004cd599af9: Status 404 returned error can't find the container with id 573324444d59ee26474b51bf1b1c8ff7f42ab58cc23642c50a1de004cd599af9
Jul 17 19:07:22.767608 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:22.767582    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl"
Jul 17 19:07:22.861186 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:22.861162    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82fa5e61-18a1-4169-8db7-277cac1368c1-maas-api-service-ca\") pod \"82fa5e61-18a1-4169-8db7-277cac1368c1\" (UID: \"82fa5e61-18a1-4169-8db7-277cac1368c1\") "
Jul 17 19:07:22.861597 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:22.861573    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82fa5e61-18a1-4169-8db7-277cac1368c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "82fa5e61-18a1-4169-8db7-277cac1368c1" (UID: "82fa5e61-18a1-4169-8db7-277cac1368c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:07:22.962047 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:22.962016    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82fa5e61-18a1-4169-8db7-277cac1368c1-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:07:23.630049 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:23.630020    2559 generic.go:356] "Generic (PLEG): container finished" podID="6f4ea9be-ddca-435e-8c65-fc641214fafa" containerID="8512f4ce8931b67b0c85ed6d92e2ad59634455759ade80cbb0cffa0d29d429f8" exitCode=7
Jul 17 19:07:23.630229 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:23.630115    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9" event={"ID":"6f4ea9be-ddca-435e-8c65-fc641214fafa","Type":"ContainerDied","Data":"8512f4ce8931b67b0c85ed6d92e2ad59634455759ade80cbb0cffa0d29d429f8"}
Jul 17 19:07:23.630229 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:23.630175    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9" event={"ID":"6f4ea9be-ddca-435e-8c65-fc641214fafa","Type":"ContainerStarted","Data":"573324444d59ee26474b51bf1b1c8ff7f42ab58cc23642c50a1de004cd599af9"}
Jul 17 19:07:23.630369 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:23.630268    2559 scope.go:117] "RemoveContainer" containerID="8512f4ce8931b67b0c85ed6d92e2ad59634455759ade80cbb0cffa0d29d429f8"
Jul 17 19:07:23.631590 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:23.631571    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl"
Jul 17 19:07:23.631709 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:23.631573    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl" event={"ID":"82fa5e61-18a1-4169-8db7-277cac1368c1","Type":"ContainerDied","Data":"f334fa2afce191ce393dcf8a07dd7e94f514732666ba4ec4da02b1e6542412c4"}
Jul 17 19:07:23.631709 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:23.631693    2559 scope.go:117] "RemoveContainer" containerID="c015b46925cb834f4c50125049f165434da1238e3e7eb53714bb81492bba1efd"
Jul 17 19:07:23.662971 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:23.662945    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl"]
Jul 17 19:07:23.667823 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:23.667801    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-9mbzl"]
Jul 17 19:07:24.125859 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:24.125780    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82fa5e61-18a1-4169-8db7-277cac1368c1" path="/var/lib/kubelet/pods/82fa5e61-18a1-4169-8db7-277cac1368c1/volumes"
Jul 17 19:07:24.638832 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:24.638797    2559 generic.go:356] "Generic (PLEG): container finished" podID="6f4ea9be-ddca-435e-8c65-fc641214fafa" containerID="0646b70d054ce0bc8771f71fa41256244adb9c9cd2a8177deac7f3a15bbcc3a9" exitCode=7
Jul 17 19:07:24.639031 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:24.638883    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9" event={"ID":"6f4ea9be-ddca-435e-8c65-fc641214fafa","Type":"ContainerDied","Data":"0646b70d054ce0bc8771f71fa41256244adb9c9cd2a8177deac7f3a15bbcc3a9"}
Jul 17 19:07:24.639031 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:24.638969    2559 scope.go:117] "RemoveContainer" containerID="8512f4ce8931b67b0c85ed6d92e2ad59634455759ade80cbb0cffa0d29d429f8"
Jul 17 19:07:24.639149 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:24.639118    2559 scope.go:117] "RemoveContainer" containerID="0646b70d054ce0bc8771f71fa41256244adb9c9cd2a8177deac7f3a15bbcc3a9"
Jul 17 19:07:24.639400 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:07:24.639367    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9_odh-ai-gateway-infra(6f4ea9be-ddca-435e-8c65-fc641214fafa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9" podUID="6f4ea9be-ddca-435e-8c65-fc641214fafa"
Jul 17 19:07:25.646926 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:25.646894    2559 scope.go:117] "RemoveContainer" containerID="0646b70d054ce0bc8771f71fa41256244adb9c9cd2a8177deac7f3a15bbcc3a9"
Jul 17 19:07:25.647343 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:07:25.647097    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9_odh-ai-gateway-infra(6f4ea9be-ddca-435e-8c65-fc641214fafa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9" podUID="6f4ea9be-ddca-435e-8c65-fc641214fafa"
Jul 17 19:07:39.120177 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:39.120098    2559 scope.go:117] "RemoveContainer" containerID="0646b70d054ce0bc8771f71fa41256244adb9c9cd2a8177deac7f3a15bbcc3a9"
Jul 17 19:07:39.725693 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:39.725662    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9" event={"ID":"6f4ea9be-ddca-435e-8c65-fc641214fafa","Type":"ContainerStarted","Data":"04744c1ea670e8f56d1a8a9d3acec3b19a121145ecdfde2aef19ea3ed9786d0c"}
Jul 17 19:07:39.743792 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:39.743750    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9" podStartSLOduration=17.743735316 podStartE2EDuration="17.743735316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:07:39.742335019 +0000 UTC m=+2070.183843709" watchObservedRunningTime="2026-07-17 19:07:39.743735316 +0000 UTC m=+2070.185244004"
Jul 17 19:07:40.733056 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:40.733022    2559 generic.go:356] "Generic (PLEG): container finished" podID="6f4ea9be-ddca-435e-8c65-fc641214fafa" containerID="04744c1ea670e8f56d1a8a9d3acec3b19a121145ecdfde2aef19ea3ed9786d0c" exitCode=7
Jul 17 19:07:40.733473 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:40.733074    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9" event={"ID":"6f4ea9be-ddca-435e-8c65-fc641214fafa","Type":"ContainerDied","Data":"04744c1ea670e8f56d1a8a9d3acec3b19a121145ecdfde2aef19ea3ed9786d0c"}
Jul 17 19:07:40.733473 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:40.733155    2559 scope.go:117] "RemoveContainer" containerID="0646b70d054ce0bc8771f71fa41256244adb9c9cd2a8177deac7f3a15bbcc3a9"
Jul 17 19:07:40.733473 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:40.733442    2559 scope.go:117] "RemoveContainer" containerID="04744c1ea670e8f56d1a8a9d3acec3b19a121145ecdfde2aef19ea3ed9786d0c"
Jul 17 19:07:40.733689 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:07:40.733663    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9_odh-ai-gateway-infra(6f4ea9be-ddca-435e-8c65-fc641214fafa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9" podUID="6f4ea9be-ddca-435e-8c65-fc641214fafa"
Jul 17 19:07:40.755588 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:40.755565    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9"]
Jul 17 19:07:41.882028 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:41.882004    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9"
Jul 17 19:07:42.026396 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:42.026332    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4ea9be-ddca-435e-8c65-fc641214fafa-maas-api-service-ca\") pod \"6f4ea9be-ddca-435e-8c65-fc641214fafa\" (UID: \"6f4ea9be-ddca-435e-8c65-fc641214fafa\") "
Jul 17 19:07:42.026648 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:42.026628    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4ea9be-ddca-435e-8c65-fc641214fafa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6f4ea9be-ddca-435e-8c65-fc641214fafa" (UID: "6f4ea9be-ddca-435e-8c65-fc641214fafa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:07:42.126973 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:42.126951    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4ea9be-ddca-435e-8c65-fc641214fafa-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:07:42.746543 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:42.746518    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9"
Jul 17 19:07:42.746715 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:42.746551    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9" event={"ID":"6f4ea9be-ddca-435e-8c65-fc641214fafa","Type":"ContainerDied","Data":"573324444d59ee26474b51bf1b1c8ff7f42ab58cc23642c50a1de004cd599af9"}
Jul 17 19:07:42.746715 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:42.746607    2559 scope.go:117] "RemoveContainer" containerID="04744c1ea670e8f56d1a8a9d3acec3b19a121145ecdfde2aef19ea3ed9786d0c"
Jul 17 19:07:42.764467 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:42.764446    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9"]
Jul 17 19:07:42.770216 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:42.770196    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-gphs9"]
Jul 17 19:07:44.126505 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:07:44.126475    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f4ea9be-ddca-435e-8c65-fc641214fafa" path="/var/lib/kubelet/pods/6f4ea9be-ddca-435e-8c65-fc641214fafa/volumes"
Jul 17 19:08:03.376834 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.376801    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp"]
Jul 17 19:08:03.377249 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.377181    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82fa5e61-18a1-4169-8db7-277cac1368c1" containerName="revoke-keys"
Jul 17 19:08:03.377249 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.377192    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="82fa5e61-18a1-4169-8db7-277cac1368c1" containerName="revoke-keys"
Jul 17 19:08:03.377249 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.377203    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f4ea9be-ddca-435e-8c65-fc641214fafa" containerName="revoke-keys"
Jul 17 19:08:03.377249 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.377209    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f4ea9be-ddca-435e-8c65-fc641214fafa" containerName="revoke-keys"
Jul 17 19:08:03.377249 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.377216    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f4ea9be-ddca-435e-8c65-fc641214fafa" containerName="revoke-keys"
Jul 17 19:08:03.377249 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.377222    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f4ea9be-ddca-435e-8c65-fc641214fafa" containerName="revoke-keys"
Jul 17 19:08:03.377249 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.377236    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f4ea9be-ddca-435e-8c65-fc641214fafa" containerName="revoke-keys"
Jul 17 19:08:03.377249 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.377241    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f4ea9be-ddca-435e-8c65-fc641214fafa" containerName="revoke-keys"
Jul 17 19:08:03.377249 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.377248    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82fa5e61-18a1-4169-8db7-277cac1368c1" containerName="revoke-keys"
Jul 17 19:08:03.377249 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.377255    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="82fa5e61-18a1-4169-8db7-277cac1368c1" containerName="revoke-keys"
Jul 17 19:08:03.377593 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.377262    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82fa5e61-18a1-4169-8db7-277cac1368c1" containerName="revoke-keys"
Jul 17 19:08:03.377593 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.377267    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="82fa5e61-18a1-4169-8db7-277cac1368c1" containerName="revoke-keys"
Jul 17 19:08:03.377593 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.377354    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="82fa5e61-18a1-4169-8db7-277cac1368c1" containerName="revoke-keys"
Jul 17 19:08:03.377593 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.377363    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f4ea9be-ddca-435e-8c65-fc641214fafa" containerName="revoke-keys"
Jul 17 19:08:03.377593 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.377370    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f4ea9be-ddca-435e-8c65-fc641214fafa" containerName="revoke-keys"
Jul 17 19:08:03.377593 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.377377    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="82fa5e61-18a1-4169-8db7-277cac1368c1" containerName="revoke-keys"
Jul 17 19:08:03.381794 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.381779    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp"
Jul 17 19:08:03.384524 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.384502    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:08:03.389019 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.388996    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp"]
Jul 17 19:08:03.493947 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.493921    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a4a84e-686f-44d8-b3bf-28e3126e7333-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp\" (UID: \"80a4a84e-686f-44d8-b3bf-28e3126e7333\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp"
Jul 17 19:08:03.594748 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.594725    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a4a84e-686f-44d8-b3bf-28e3126e7333-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp\" (UID: \"80a4a84e-686f-44d8-b3bf-28e3126e7333\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp"
Jul 17 19:08:03.595284 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.595266    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a4a84e-686f-44d8-b3bf-28e3126e7333-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp\" (UID: \"80a4a84e-686f-44d8-b3bf-28e3126e7333\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp"
Jul 17 19:08:03.693231 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.693184    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp"
Jul 17 19:08:03.827044 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.827011    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp"]
Jul 17 19:08:03.828253 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:08:03.828223    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a4a84e_686f_44d8_b3bf_28e3126e7333.slice/crio-d429b1f5e2cb12cc85a5d8c0f71ac4ca5991f8ca40e163ee563fb21ea6fd2e3e WatchSource:0}: Error finding container d429b1f5e2cb12cc85a5d8c0f71ac4ca5991f8ca40e163ee563fb21ea6fd2e3e: Status 404 returned error can't find the container with id d429b1f5e2cb12cc85a5d8c0f71ac4ca5991f8ca40e163ee563fb21ea6fd2e3e
Jul 17 19:08:03.866484 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:03.866450    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp" event={"ID":"80a4a84e-686f-44d8-b3bf-28e3126e7333","Type":"ContainerStarted","Data":"d429b1f5e2cb12cc85a5d8c0f71ac4ca5991f8ca40e163ee563fb21ea6fd2e3e"}
Jul 17 19:08:04.873383 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:04.873347    2559 generic.go:356] "Generic (PLEG): container finished" podID="80a4a84e-686f-44d8-b3bf-28e3126e7333" containerID="c2aaa8e77d5af04bd0c4427e4c878ea909945cf314786b0dfd259631a46270bf" exitCode=6
Jul 17 19:08:04.873768 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:04.873433    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp" event={"ID":"80a4a84e-686f-44d8-b3bf-28e3126e7333","Type":"ContainerDied","Data":"c2aaa8e77d5af04bd0c4427e4c878ea909945cf314786b0dfd259631a46270bf"}
Jul 17 19:08:04.873768 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:04.873577    2559 scope.go:117] "RemoveContainer" containerID="c2aaa8e77d5af04bd0c4427e4c878ea909945cf314786b0dfd259631a46270bf"
Jul 17 19:08:05.881263 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:05.881231    2559 generic.go:356] "Generic (PLEG): container finished" podID="80a4a84e-686f-44d8-b3bf-28e3126e7333" containerID="40470c5819a62665ee0f83f5cfbe0d92ef6aaaa6a1e1e93228d9322e3b62c420" exitCode=6
Jul 17 19:08:05.881707 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:05.881338    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp" event={"ID":"80a4a84e-686f-44d8-b3bf-28e3126e7333","Type":"ContainerDied","Data":"40470c5819a62665ee0f83f5cfbe0d92ef6aaaa6a1e1e93228d9322e3b62c420"}
Jul 17 19:08:05.881707 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:05.881440    2559 scope.go:117] "RemoveContainer" containerID="c2aaa8e77d5af04bd0c4427e4c878ea909945cf314786b0dfd259631a46270bf"
Jul 17 19:08:05.881707 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:05.881481    2559 scope.go:117] "RemoveContainer" containerID="40470c5819a62665ee0f83f5cfbe0d92ef6aaaa6a1e1e93228d9322e3b62c420"
Jul 17 19:08:05.881707 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:08:05.881646    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp_odh-ai-gateway-infra(80a4a84e-686f-44d8-b3bf-28e3126e7333)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp" podUID="80a4a84e-686f-44d8-b3bf-28e3126e7333"
Jul 17 19:08:06.888914 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:06.888891    2559 scope.go:117] "RemoveContainer" containerID="40470c5819a62665ee0f83f5cfbe0d92ef6aaaa6a1e1e93228d9322e3b62c420"
Jul 17 19:08:06.889289 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:08:06.889044    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp_odh-ai-gateway-infra(80a4a84e-686f-44d8-b3bf-28e3126e7333)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp" podUID="80a4a84e-686f-44d8-b3bf-28e3126e7333"
Jul 17 19:08:10.273060 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:10.273036    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 19:08:10.277458 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:10.277439    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 19:08:10.280688 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:10.280666    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 19:08:10.285340 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:10.285301    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 19:08:15.197611 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:15.197576    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts"]
Jul 17 19:08:15.198035 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:15.198021    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="82fa5e61-18a1-4169-8db7-277cac1368c1" containerName="revoke-keys"
Jul 17 19:08:15.198076 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:15.198040    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f4ea9be-ddca-435e-8c65-fc641214fafa" containerName="revoke-keys"
Jul 17 19:08:15.200887 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:15.200873    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts"
Jul 17 19:08:15.207439 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:15.207410    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts"]
Jul 17 19:08:15.295431 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:15.295399    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e882ec18-02da-49a2-ac2d-1d22cf3bb3d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts\" (UID: \"e882ec18-02da-49a2-ac2d-1d22cf3bb3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts"
Jul 17 19:08:15.396508 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:15.396481    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e882ec18-02da-49a2-ac2d-1d22cf3bb3d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts\" (UID: \"e882ec18-02da-49a2-ac2d-1d22cf3bb3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts"
Jul 17 19:08:15.397055 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:15.397038    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e882ec18-02da-49a2-ac2d-1d22cf3bb3d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts\" (UID: \"e882ec18-02da-49a2-ac2d-1d22cf3bb3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts"
Jul 17 19:08:15.512413 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:15.512357    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts"
Jul 17 19:08:15.643577 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:15.643553    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts"]
Jul 17 19:08:15.645538 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:08:15.645511    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode882ec18_02da_49a2_ac2d_1d22cf3bb3d1.slice/crio-7b2eee1971040e2954a22e3941db24aa78d2be8b7f26fe3ad6c40d3910340d51 WatchSource:0}: Error finding container 7b2eee1971040e2954a22e3941db24aa78d2be8b7f26fe3ad6c40d3910340d51: Status 404 returned error can't find the container with id 7b2eee1971040e2954a22e3941db24aa78d2be8b7f26fe3ad6c40d3910340d51
Jul 17 19:08:15.943573 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:15.943541    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts" event={"ID":"e882ec18-02da-49a2-ac2d-1d22cf3bb3d1","Type":"ContainerStarted","Data":"7b2eee1971040e2954a22e3941db24aa78d2be8b7f26fe3ad6c40d3910340d51"}
Jul 17 19:08:16.950798 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:16.950766    2559 generic.go:356] "Generic (PLEG): container finished" podID="e882ec18-02da-49a2-ac2d-1d22cf3bb3d1" containerID="e4c6e7121c8017264c698660d8f99675ee8c7ecb8afa713c0eec1d6a1c29b72c" exitCode=6
Jul 17 19:08:16.950798 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:16.950811    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts" event={"ID":"e882ec18-02da-49a2-ac2d-1d22cf3bb3d1","Type":"ContainerDied","Data":"e4c6e7121c8017264c698660d8f99675ee8c7ecb8afa713c0eec1d6a1c29b72c"}
Jul 17 19:08:16.951262 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:16.951017    2559 scope.go:117] "RemoveContainer" containerID="e4c6e7121c8017264c698660d8f99675ee8c7ecb8afa713c0eec1d6a1c29b72c"
Jul 17 19:08:17.958592 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:17.958560    2559 generic.go:356] "Generic (PLEG): container finished" podID="e882ec18-02da-49a2-ac2d-1d22cf3bb3d1" containerID="3a7896cf01b84dd008951cb3cba47b8840e0e7359b69753c76d0b6e8678e7c97" exitCode=6
Jul 17 19:08:17.959037 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:17.958609    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts" event={"ID":"e882ec18-02da-49a2-ac2d-1d22cf3bb3d1","Type":"ContainerDied","Data":"3a7896cf01b84dd008951cb3cba47b8840e0e7359b69753c76d0b6e8678e7c97"}
Jul 17 19:08:17.959037 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:17.958661    2559 scope.go:117] "RemoveContainer" containerID="e4c6e7121c8017264c698660d8f99675ee8c7ecb8afa713c0eec1d6a1c29b72c"
Jul 17 19:08:17.959037 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:17.958819    2559 scope.go:117] "RemoveContainer" containerID="3a7896cf01b84dd008951cb3cba47b8840e0e7359b69753c76d0b6e8678e7c97"
Jul 17 19:08:17.959163 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:08:17.959038    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts_odh-ai-gateway-infra(e882ec18-02da-49a2-ac2d-1d22cf3bb3d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts" podUID="e882ec18-02da-49a2-ac2d-1d22cf3bb3d1"
Jul 17 19:08:18.966420 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:18.966391    2559 scope.go:117] "RemoveContainer" containerID="3a7896cf01b84dd008951cb3cba47b8840e0e7359b69753c76d0b6e8678e7c97"
Jul 17 19:08:18.966882 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:08:18.966557    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts_odh-ai-gateway-infra(e882ec18-02da-49a2-ac2d-1d22cf3bb3d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts" podUID="e882ec18-02da-49a2-ac2d-1d22cf3bb3d1"
Jul 17 19:08:19.119487 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:19.119460    2559 scope.go:117] "RemoveContainer" containerID="40470c5819a62665ee0f83f5cfbe0d92ef6aaaa6a1e1e93228d9322e3b62c420"
Jul 17 19:08:19.974132 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:19.974105    2559 generic.go:356] "Generic (PLEG): container finished" podID="80a4a84e-686f-44d8-b3bf-28e3126e7333" containerID="5bacc86409f31c7629574a6f760feefa4e75ed86619eeac8e4936544e9651aab" exitCode=6
Jul 17 19:08:19.974547 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:19.974174    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp" event={"ID":"80a4a84e-686f-44d8-b3bf-28e3126e7333","Type":"ContainerDied","Data":"5bacc86409f31c7629574a6f760feefa4e75ed86619eeac8e4936544e9651aab"}
Jul 17 19:08:19.974547 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:19.974237    2559 scope.go:117] "RemoveContainer" containerID="40470c5819a62665ee0f83f5cfbe0d92ef6aaaa6a1e1e93228d9322e3b62c420"
Jul 17 19:08:19.974547 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:19.974528    2559 scope.go:117] "RemoveContainer" containerID="5bacc86409f31c7629574a6f760feefa4e75ed86619eeac8e4936544e9651aab"
Jul 17 19:08:19.974719 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:08:19.974698    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp_odh-ai-gateway-infra(80a4a84e-686f-44d8-b3bf-28e3126e7333)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp" podUID="80a4a84e-686f-44d8-b3bf-28e3126e7333"
Jul 17 19:08:21.002231 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:21.002199    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp"]
Jul 17 19:08:21.138553 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:21.138531    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp"
Jul 17 19:08:21.243795 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:21.243759    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a4a84e-686f-44d8-b3bf-28e3126e7333-maas-api-service-ca\") pod \"80a4a84e-686f-44d8-b3bf-28e3126e7333\" (UID: \"80a4a84e-686f-44d8-b3bf-28e3126e7333\") "
Jul 17 19:08:21.244119 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:21.244094    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a4a84e-686f-44d8-b3bf-28e3126e7333-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "80a4a84e-686f-44d8-b3bf-28e3126e7333" (UID: "80a4a84e-686f-44d8-b3bf-28e3126e7333"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:08:21.344997 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:21.344973    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a4a84e-686f-44d8-b3bf-28e3126e7333-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:08:21.988205 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:21.988179    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp"
Jul 17 19:08:21.988415 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:21.988204    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp" event={"ID":"80a4a84e-686f-44d8-b3bf-28e3126e7333","Type":"ContainerDied","Data":"d429b1f5e2cb12cc85a5d8c0f71ac4ca5991f8ca40e163ee563fb21ea6fd2e3e"}
Jul 17 19:08:21.988415 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:21.988262    2559 scope.go:117] "RemoveContainer" containerID="5bacc86409f31c7629574a6f760feefa4e75ed86619eeac8e4936544e9651aab"
Jul 17 19:08:22.010940 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:22.010917    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp"]
Jul 17 19:08:22.018407 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:22.018380    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-77skp"]
Jul 17 19:08:22.125805 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:22.125777    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80a4a84e-686f-44d8-b3bf-28e3126e7333" path="/var/lib/kubelet/pods/80a4a84e-686f-44d8-b3bf-28e3126e7333/volumes"
Jul 17 19:08:22.254370 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:22.254289    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl"]
Jul 17 19:08:22.254910 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:22.254890    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80a4a84e-686f-44d8-b3bf-28e3126e7333" containerName="revoke-keys"
Jul 17 19:08:22.254951 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:22.254916    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="80a4a84e-686f-44d8-b3bf-28e3126e7333" containerName="revoke-keys"
Jul 17 19:08:22.254951 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:22.254931    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80a4a84e-686f-44d8-b3bf-28e3126e7333" containerName="revoke-keys"
Jul 17 19:08:22.254951 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:22.254938    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="80a4a84e-686f-44d8-b3bf-28e3126e7333" containerName="revoke-keys"
Jul 17 19:08:22.255062 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:22.255050    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="80a4a84e-686f-44d8-b3bf-28e3126e7333" containerName="revoke-keys"
Jul 17 19:08:22.255100 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:22.255069    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="80a4a84e-686f-44d8-b3bf-28e3126e7333" containerName="revoke-keys"
Jul 17 19:08:22.255100 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:22.255079    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="80a4a84e-686f-44d8-b3bf-28e3126e7333" containerName="revoke-keys"
Jul 17 19:08:22.260146 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:22.259903    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl"
Jul 17 19:08:22.262986 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:22.262959    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl"]
Jul 17 19:08:22.353935 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:22.353909    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1324d9e7-28ca-41ea-8cb2-4b83503cc43c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl\" (UID: \"1324d9e7-28ca-41ea-8cb2-4b83503cc43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl"
Jul 17 19:08:22.454647 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:22.454624    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1324d9e7-28ca-41ea-8cb2-4b83503cc43c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl\" (UID: \"1324d9e7-28ca-41ea-8cb2-4b83503cc43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl"
Jul 17 19:08:22.455137 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:22.455119    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1324d9e7-28ca-41ea-8cb2-4b83503cc43c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl\" (UID: \"1324d9e7-28ca-41ea-8cb2-4b83503cc43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl"
Jul 17 19:08:22.572901 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:22.572876    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl"
Jul 17 19:08:22.709617 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:22.709595    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl"]
Jul 17 19:08:22.711253 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:08:22.711228    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1324d9e7_28ca_41ea_8cb2_4b83503cc43c.slice/crio-5c4aa15cec7aa6b20b27eaff55bb824c4dbae1698e38e5add1973d0e244e4d32 WatchSource:0}: Error finding container 5c4aa15cec7aa6b20b27eaff55bb824c4dbae1698e38e5add1973d0e244e4d32: Status 404 returned error can't find the container with id 5c4aa15cec7aa6b20b27eaff55bb824c4dbae1698e38e5add1973d0e244e4d32
Jul 17 19:08:22.995732 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:22.995703    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl" event={"ID":"1324d9e7-28ca-41ea-8cb2-4b83503cc43c","Type":"ContainerStarted","Data":"2a77131fd90797088c429d1f9a3209f22afad5a060eead25bb53b802e3270285"}
Jul 17 19:08:22.995850 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:22.995764    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl" event={"ID":"1324d9e7-28ca-41ea-8cb2-4b83503cc43c","Type":"ContainerStarted","Data":"5c4aa15cec7aa6b20b27eaff55bb824c4dbae1698e38e5add1973d0e244e4d32"}
Jul 17 19:08:23.013058 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:23.013016    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl" podStartSLOduration=1.013001038 podStartE2EDuration="1.013001038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:08:23.012264062 +0000 UTC m=+2113.453772764" watchObservedRunningTime="2026-07-17 19:08:23.013001038 +0000 UTC m=+2113.454509727"
Jul 17 19:08:24.003565 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:24.003530    2559 generic.go:356] "Generic (PLEG): container finished" podID="1324d9e7-28ca-41ea-8cb2-4b83503cc43c" containerID="2a77131fd90797088c429d1f9a3209f22afad5a060eead25bb53b802e3270285" exitCode=7
Jul 17 19:08:24.003731 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:24.003612    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl" event={"ID":"1324d9e7-28ca-41ea-8cb2-4b83503cc43c","Type":"ContainerDied","Data":"2a77131fd90797088c429d1f9a3209f22afad5a060eead25bb53b802e3270285"}
Jul 17 19:08:24.003785 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:24.003766    2559 scope.go:117] "RemoveContainer" containerID="2a77131fd90797088c429d1f9a3209f22afad5a060eead25bb53b802e3270285"
Jul 17 19:08:25.010892 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:25.010802    2559 generic.go:356] "Generic (PLEG): container finished" podID="1324d9e7-28ca-41ea-8cb2-4b83503cc43c" containerID="817a5fb7677494f16dc37be1fa70c7ab366b4b6cf6731d1faa597853d1ed5507" exitCode=7
Jul 17 19:08:25.010892 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:25.010866    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl" event={"ID":"1324d9e7-28ca-41ea-8cb2-4b83503cc43c","Type":"ContainerDied","Data":"817a5fb7677494f16dc37be1fa70c7ab366b4b6cf6731d1faa597853d1ed5507"}
Jul 17 19:08:25.011377 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:25.010919    2559 scope.go:117] "RemoveContainer" containerID="2a77131fd90797088c429d1f9a3209f22afad5a060eead25bb53b802e3270285"
Jul 17 19:08:25.011377 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:25.011045    2559 scope.go:117] "RemoveContainer" containerID="817a5fb7677494f16dc37be1fa70c7ab366b4b6cf6731d1faa597853d1ed5507"
Jul 17 19:08:25.011377 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:08:25.011280    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl_odh-ai-gateway-infra(1324d9e7-28ca-41ea-8cb2-4b83503cc43c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl" podUID="1324d9e7-28ca-41ea-8cb2-4b83503cc43c"
Jul 17 19:08:26.018517 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:26.018491    2559 scope.go:117] "RemoveContainer" containerID="817a5fb7677494f16dc37be1fa70c7ab366b4b6cf6731d1faa597853d1ed5507"
Jul 17 19:08:26.018936 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:08:26.018679    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl_odh-ai-gateway-infra(1324d9e7-28ca-41ea-8cb2-4b83503cc43c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl" podUID="1324d9e7-28ca-41ea-8cb2-4b83503cc43c"
Jul 17 19:08:31.120337 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:31.120281    2559 scope.go:117] "RemoveContainer" containerID="3a7896cf01b84dd008951cb3cba47b8840e0e7359b69753c76d0b6e8678e7c97"
Jul 17 19:08:32.053964 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:32.053934    2559 generic.go:356] "Generic (PLEG): container finished" podID="e882ec18-02da-49a2-ac2d-1d22cf3bb3d1" containerID="c4ead9cdec67230ba0b7ddc3d995a0432152217119dc1904dbe59e86819c2337" exitCode=6
Jul 17 19:08:32.054136 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:32.053979    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts" event={"ID":"e882ec18-02da-49a2-ac2d-1d22cf3bb3d1","Type":"ContainerDied","Data":"c4ead9cdec67230ba0b7ddc3d995a0432152217119dc1904dbe59e86819c2337"}
Jul 17 19:08:32.054136 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:32.054052    2559 scope.go:117] "RemoveContainer" containerID="3a7896cf01b84dd008951cb3cba47b8840e0e7359b69753c76d0b6e8678e7c97"
Jul 17 19:08:32.054364 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:32.054338    2559 scope.go:117] "RemoveContainer" containerID="c4ead9cdec67230ba0b7ddc3d995a0432152217119dc1904dbe59e86819c2337"
Jul 17 19:08:32.054606 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:08:32.054587    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts_odh-ai-gateway-infra(e882ec18-02da-49a2-ac2d-1d22cf3bb3d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts" podUID="e882ec18-02da-49a2-ac2d-1d22cf3bb3d1"
Jul 17 19:08:33.082514 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:33.082475    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts"]
Jul 17 19:08:33.219088 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:33.219064    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts"
Jul 17 19:08:33.350529 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:33.350455    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e882ec18-02da-49a2-ac2d-1d22cf3bb3d1-maas-api-service-ca\") pod \"e882ec18-02da-49a2-ac2d-1d22cf3bb3d1\" (UID: \"e882ec18-02da-49a2-ac2d-1d22cf3bb3d1\") "
Jul 17 19:08:33.350781 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:33.350759    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e882ec18-02da-49a2-ac2d-1d22cf3bb3d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e882ec18-02da-49a2-ac2d-1d22cf3bb3d1" (UID: "e882ec18-02da-49a2-ac2d-1d22cf3bb3d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:08:33.451475 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:33.451450    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e882ec18-02da-49a2-ac2d-1d22cf3bb3d1-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:08:34.067935 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:34.067907    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts"
Jul 17 19:08:34.068129 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:34.067944    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts" event={"ID":"e882ec18-02da-49a2-ac2d-1d22cf3bb3d1","Type":"ContainerDied","Data":"7b2eee1971040e2954a22e3941db24aa78d2be8b7f26fe3ad6c40d3910340d51"}
Jul 17 19:08:34.068129 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:34.068005    2559 scope.go:117] "RemoveContainer" containerID="c4ead9cdec67230ba0b7ddc3d995a0432152217119dc1904dbe59e86819c2337"
Jul 17 19:08:34.090936 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:34.090913    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts"]
Jul 17 19:08:34.098352 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:34.098330    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-8bmts"]
Jul 17 19:08:34.125272 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:34.125245    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e882ec18-02da-49a2-ac2d-1d22cf3bb3d1" path="/var/lib/kubelet/pods/e882ec18-02da-49a2-ac2d-1d22cf3bb3d1/volumes"
Jul 17 19:08:40.122253 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:40.122215    2559 scope.go:117] "RemoveContainer" containerID="817a5fb7677494f16dc37be1fa70c7ab366b4b6cf6731d1faa597853d1ed5507"
Jul 17 19:08:41.108905 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:41.108873    2559 generic.go:356] "Generic (PLEG): container finished" podID="1324d9e7-28ca-41ea-8cb2-4b83503cc43c" containerID="1c57e4ef14b64cb89ffdf8ec989899845cb67e2054b26c3574893198d41efbba" exitCode=7
Jul 17 19:08:41.109064 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:41.108928    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl" event={"ID":"1324d9e7-28ca-41ea-8cb2-4b83503cc43c","Type":"ContainerDied","Data":"1c57e4ef14b64cb89ffdf8ec989899845cb67e2054b26c3574893198d41efbba"}
Jul 17 19:08:41.109064 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:41.109002    2559 scope.go:117] "RemoveContainer" containerID="817a5fb7677494f16dc37be1fa70c7ab366b4b6cf6731d1faa597853d1ed5507"
Jul 17 19:08:41.109349 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:41.109322    2559 scope.go:117] "RemoveContainer" containerID="1c57e4ef14b64cb89ffdf8ec989899845cb67e2054b26c3574893198d41efbba"
Jul 17 19:08:41.109553 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:08:41.109529    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl_odh-ai-gateway-infra(1324d9e7-28ca-41ea-8cb2-4b83503cc43c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl" podUID="1324d9e7-28ca-41ea-8cb2-4b83503cc43c"
Jul 17 19:08:42.135905 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:42.135877    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl"]
Jul 17 19:08:42.270216 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:42.270193    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl"
Jul 17 19:08:42.322376 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:42.322349    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1324d9e7-28ca-41ea-8cb2-4b83503cc43c-maas-api-service-ca\") pod \"1324d9e7-28ca-41ea-8cb2-4b83503cc43c\" (UID: \"1324d9e7-28ca-41ea-8cb2-4b83503cc43c\") "
Jul 17 19:08:42.322718 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:42.322697    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1324d9e7-28ca-41ea-8cb2-4b83503cc43c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1324d9e7-28ca-41ea-8cb2-4b83503cc43c" (UID: "1324d9e7-28ca-41ea-8cb2-4b83503cc43c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:08:42.423653 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:42.423601    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1324d9e7-28ca-41ea-8cb2-4b83503cc43c-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:08:43.123181 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:43.123156    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl"
Jul 17 19:08:43.123366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:43.123197    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl" event={"ID":"1324d9e7-28ca-41ea-8cb2-4b83503cc43c","Type":"ContainerDied","Data":"5c4aa15cec7aa6b20b27eaff55bb824c4dbae1698e38e5add1973d0e244e4d32"}
Jul 17 19:08:43.123366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:43.123281    2559 scope.go:117] "RemoveContainer" containerID="1c57e4ef14b64cb89ffdf8ec989899845cb67e2054b26c3574893198d41efbba"
Jul 17 19:08:43.147925 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:43.147900    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl"]
Jul 17 19:08:43.154153 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:43.154130    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-mt4bl"]
Jul 17 19:08:44.125508 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:44.125478    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1324d9e7-28ca-41ea-8cb2-4b83503cc43c" path="/var/lib/kubelet/pods/1324d9e7-28ca-41ea-8cb2-4b83503cc43c/volumes"
Jul 17 19:08:53.423093 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.423058    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx"]
Jul 17 19:08:53.423589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.423433    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e882ec18-02da-49a2-ac2d-1d22cf3bb3d1" containerName="revoke-keys"
Jul 17 19:08:53.423589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.423444    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e882ec18-02da-49a2-ac2d-1d22cf3bb3d1" containerName="revoke-keys"
Jul 17 19:08:53.423589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.423452    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80a4a84e-686f-44d8-b3bf-28e3126e7333" containerName="revoke-keys"
Jul 17 19:08:53.423589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.423458    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="80a4a84e-686f-44d8-b3bf-28e3126e7333" containerName="revoke-keys"
Jul 17 19:08:53.423589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.423466    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1324d9e7-28ca-41ea-8cb2-4b83503cc43c" containerName="revoke-keys"
Jul 17 19:08:53.423589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.423471    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1324d9e7-28ca-41ea-8cb2-4b83503cc43c" containerName="revoke-keys"
Jul 17 19:08:53.423589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.423480    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e882ec18-02da-49a2-ac2d-1d22cf3bb3d1" containerName="revoke-keys"
Jul 17 19:08:53.423589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.423485    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e882ec18-02da-49a2-ac2d-1d22cf3bb3d1" containerName="revoke-keys"
Jul 17 19:08:53.423589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.423491    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1324d9e7-28ca-41ea-8cb2-4b83503cc43c" containerName="revoke-keys"
Jul 17 19:08:53.423589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.423497    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1324d9e7-28ca-41ea-8cb2-4b83503cc43c" containerName="revoke-keys"
Jul 17 19:08:53.423589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.423561    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1324d9e7-28ca-41ea-8cb2-4b83503cc43c" containerName="revoke-keys"
Jul 17 19:08:53.423589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.423571    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e882ec18-02da-49a2-ac2d-1d22cf3bb3d1" containerName="revoke-keys"
Jul 17 19:08:53.423589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.423577    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e882ec18-02da-49a2-ac2d-1d22cf3bb3d1" containerName="revoke-keys"
Jul 17 19:08:53.423589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.423582    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1324d9e7-28ca-41ea-8cb2-4b83503cc43c" containerName="revoke-keys"
Jul 17 19:08:53.423589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.423588    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e882ec18-02da-49a2-ac2d-1d22cf3bb3d1" containerName="revoke-keys"
Jul 17 19:08:53.427748 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.427731    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx"
Jul 17 19:08:53.430396 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.430371    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:08:53.433806 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.433784    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx"]
Jul 17 19:08:53.511960 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.511932    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0271789-29c5-492c-b5fa-ee545a323fb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx\" (UID: \"c0271789-29c5-492c-b5fa-ee545a323fb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx"
Jul 17 19:08:53.612834 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.612798    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0271789-29c5-492c-b5fa-ee545a323fb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx\" (UID: \"c0271789-29c5-492c-b5fa-ee545a323fb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx"
Jul 17 19:08:53.613454 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.613433    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0271789-29c5-492c-b5fa-ee545a323fb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx\" (UID: \"c0271789-29c5-492c-b5fa-ee545a323fb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx"
Jul 17 19:08:53.740241 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.740155    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx"
Jul 17 19:08:53.877811 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:53.877785    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx"]
Jul 17 19:08:53.880177 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:08:53.880153    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0271789_29c5_492c_b5fa_ee545a323fb5.slice/crio-3daed345de1d127498a2bf8dd563bdff97f103ba520a57db461d35ebe9b5f86f WatchSource:0}: Error finding container 3daed345de1d127498a2bf8dd563bdff97f103ba520a57db461d35ebe9b5f86f: Status 404 returned error can't find the container with id 3daed345de1d127498a2bf8dd563bdff97f103ba520a57db461d35ebe9b5f86f
Jul 17 19:08:54.185198 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:54.185166    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx" event={"ID":"c0271789-29c5-492c-b5fa-ee545a323fb5","Type":"ContainerStarted","Data":"c68042264049f137aa24bf58f43a43bd8f291a226ad1254d863fbcb7e5e39f98"}
Jul 17 19:08:54.185321 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:54.185218    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx" event={"ID":"c0271789-29c5-492c-b5fa-ee545a323fb5","Type":"ContainerStarted","Data":"3daed345de1d127498a2bf8dd563bdff97f103ba520a57db461d35ebe9b5f86f"}
Jul 17 19:08:54.218115 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:54.218071    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx" podStartSLOduration=1.2180576140000001 podStartE2EDuration="1.218057614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:08:54.21798198 +0000 UTC m=+2144.659490669" watchObservedRunningTime="2026-07-17 19:08:54.218057614 +0000 UTC m=+2144.659566303"
Jul 17 19:08:55.192828 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:55.192792    2559 generic.go:356] "Generic (PLEG): container finished" podID="c0271789-29c5-492c-b5fa-ee545a323fb5" containerID="c68042264049f137aa24bf58f43a43bd8f291a226ad1254d863fbcb7e5e39f98" exitCode=6
Jul 17 19:08:55.193398 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:55.192858    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx" event={"ID":"c0271789-29c5-492c-b5fa-ee545a323fb5","Type":"ContainerDied","Data":"c68042264049f137aa24bf58f43a43bd8f291a226ad1254d863fbcb7e5e39f98"}
Jul 17 19:08:55.193398 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:55.192991    2559 scope.go:117] "RemoveContainer" containerID="c68042264049f137aa24bf58f43a43bd8f291a226ad1254d863fbcb7e5e39f98"
Jul 17 19:08:56.204429 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:56.204397    2559 generic.go:356] "Generic (PLEG): container finished" podID="c0271789-29c5-492c-b5fa-ee545a323fb5" containerID="356561bd12afc3c1dab6c347e49d03ad3b47e5c284bc284c79feec04c18ed0c0" exitCode=6
Jul 17 19:08:56.204933 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:56.204480    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx" event={"ID":"c0271789-29c5-492c-b5fa-ee545a323fb5","Type":"ContainerDied","Data":"356561bd12afc3c1dab6c347e49d03ad3b47e5c284bc284c79feec04c18ed0c0"}
Jul 17 19:08:56.204933 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:56.204571    2559 scope.go:117] "RemoveContainer" containerID="c68042264049f137aa24bf58f43a43bd8f291a226ad1254d863fbcb7e5e39f98"
Jul 17 19:08:56.204933 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:56.204711    2559 scope.go:117] "RemoveContainer" containerID="356561bd12afc3c1dab6c347e49d03ad3b47e5c284bc284c79feec04c18ed0c0"
Jul 17 19:08:56.205067 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:08:56.204928    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx_odh-ai-gateway-infra(c0271789-29c5-492c-b5fa-ee545a323fb5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx" podUID="c0271789-29c5-492c-b5fa-ee545a323fb5"
Jul 17 19:08:57.212118 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:08:57.212090    2559 scope.go:117] "RemoveContainer" containerID="356561bd12afc3c1dab6c347e49d03ad3b47e5c284bc284c79feec04c18ed0c0"
Jul 17 19:08:57.212554 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:08:57.212259    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx_odh-ai-gateway-infra(c0271789-29c5-492c-b5fa-ee545a323fb5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx" podUID="c0271789-29c5-492c-b5fa-ee545a323fb5"
Jul 17 19:09:05.237190 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:05.237109    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg"]
Jul 17 19:09:05.237563 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:05.237494    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e882ec18-02da-49a2-ac2d-1d22cf3bb3d1" containerName="revoke-keys"
Jul 17 19:09:05.237563 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:05.237505    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e882ec18-02da-49a2-ac2d-1d22cf3bb3d1" containerName="revoke-keys"
Jul 17 19:09:05.237563 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:05.237512    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1324d9e7-28ca-41ea-8cb2-4b83503cc43c" containerName="revoke-keys"
Jul 17 19:09:05.237563 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:05.237518    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1324d9e7-28ca-41ea-8cb2-4b83503cc43c" containerName="revoke-keys"
Jul 17 19:09:05.237789 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:05.237591    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1324d9e7-28ca-41ea-8cb2-4b83503cc43c" containerName="revoke-keys"
Jul 17 19:09:05.241946 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:05.241929    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg"
Jul 17 19:09:05.250201 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:05.250177    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg"]
Jul 17 19:09:05.311300 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:05.311271    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86b76a98-c448-4162-bb65-768d10f8c60b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg\" (UID: \"86b76a98-c448-4162-bb65-768d10f8c60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg"
Jul 17 19:09:05.412522 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:05.412496    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86b76a98-c448-4162-bb65-768d10f8c60b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg\" (UID: \"86b76a98-c448-4162-bb65-768d10f8c60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg"
Jul 17 19:09:05.413061 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:05.413043    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86b76a98-c448-4162-bb65-768d10f8c60b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg\" (UID: \"86b76a98-c448-4162-bb65-768d10f8c60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg"
Jul 17 19:09:05.554559 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:05.554477    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg"
Jul 17 19:09:05.687528 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:05.687500    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg"]
Jul 17 19:09:05.689024 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:09:05.688998    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b76a98_c448_4162_bb65_768d10f8c60b.slice/crio-b04363b4cc75a451885aa61670cb0d9014da67147d250e71bfe383b7f63c345a WatchSource:0}: Error finding container b04363b4cc75a451885aa61670cb0d9014da67147d250e71bfe383b7f63c345a: Status 404 returned error can't find the container with id b04363b4cc75a451885aa61670cb0d9014da67147d250e71bfe383b7f63c345a
Jul 17 19:09:06.262582 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:06.262502    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg" event={"ID":"86b76a98-c448-4162-bb65-768d10f8c60b","Type":"ContainerStarted","Data":"dac03b059496c86c8b8a4398a26d473276a07065b9aa6abc176ae43197354901"}
Jul 17 19:09:06.262582 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:06.262559    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg" event={"ID":"86b76a98-c448-4162-bb65-768d10f8c60b","Type":"ContainerStarted","Data":"b04363b4cc75a451885aa61670cb0d9014da67147d250e71bfe383b7f63c345a"}
Jul 17 19:09:06.279703 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:06.279660    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg" podStartSLOduration=1.279647136 podStartE2EDuration="1.279647136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:09:06.277357869 +0000 UTC m=+2156.718866559" watchObservedRunningTime="2026-07-17 19:09:06.279647136 +0000 UTC m=+2156.721155824"
Jul 17 19:09:06.469552 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:09:06.469526    2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b76a98_c448_4162_bb65_768d10f8c60b.slice/crio-conmon-dac03b059496c86c8b8a4398a26d473276a07065b9aa6abc176ae43197354901.scope\": RecentStats: unable to find data in memory cache]"
Jul 17 19:09:07.269743 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:07.269707    2559 generic.go:356] "Generic (PLEG): container finished" podID="86b76a98-c448-4162-bb65-768d10f8c60b" containerID="dac03b059496c86c8b8a4398a26d473276a07065b9aa6abc176ae43197354901" exitCode=6
Jul 17 19:09:07.270212 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:07.269792    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg" event={"ID":"86b76a98-c448-4162-bb65-768d10f8c60b","Type":"ContainerDied","Data":"dac03b059496c86c8b8a4398a26d473276a07065b9aa6abc176ae43197354901"}
Jul 17 19:09:07.270212 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:07.269983    2559 scope.go:117] "RemoveContainer" containerID="dac03b059496c86c8b8a4398a26d473276a07065b9aa6abc176ae43197354901"
Jul 17 19:09:08.120173 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:08.120087    2559 scope.go:117] "RemoveContainer" containerID="356561bd12afc3c1dab6c347e49d03ad3b47e5c284bc284c79feec04c18ed0c0"
Jul 17 19:09:08.277653 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:08.277627    2559 generic.go:356] "Generic (PLEG): container finished" podID="86b76a98-c448-4162-bb65-768d10f8c60b" containerID="4f01a94b16f18968b364b07426096c79ead29e725da0e89fba568b736b0ae63d" exitCode=6
Jul 17 19:09:08.278041 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:08.277676    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg" event={"ID":"86b76a98-c448-4162-bb65-768d10f8c60b","Type":"ContainerDied","Data":"4f01a94b16f18968b364b07426096c79ead29e725da0e89fba568b736b0ae63d"}
Jul 17 19:09:08.278041 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:08.277735    2559 scope.go:117] "RemoveContainer" containerID="dac03b059496c86c8b8a4398a26d473276a07065b9aa6abc176ae43197354901"
Jul 17 19:09:08.278041 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:08.277937    2559 scope.go:117] "RemoveContainer" containerID="4f01a94b16f18968b364b07426096c79ead29e725da0e89fba568b736b0ae63d"
Jul 17 19:09:08.278198 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:09:08.278161    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg_odh-ai-gateway-infra(86b76a98-c448-4162-bb65-768d10f8c60b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg" podUID="86b76a98-c448-4162-bb65-768d10f8c60b"
Jul 17 19:09:09.285115 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:09.285074    2559 scope.go:117] "RemoveContainer" containerID="4f01a94b16f18968b364b07426096c79ead29e725da0e89fba568b736b0ae63d"
Jul 17 19:09:09.285554 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:09:09.285264    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg_odh-ai-gateway-infra(86b76a98-c448-4162-bb65-768d10f8c60b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg" podUID="86b76a98-c448-4162-bb65-768d10f8c60b"
Jul 17 19:09:09.287077 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:09.287051    2559 generic.go:356] "Generic (PLEG): container finished" podID="c0271789-29c5-492c-b5fa-ee545a323fb5" containerID="eedeb3fa85b504984d8d736ed94b3789d7048c1d594115f27e141d5191c93a41" exitCode=6
Jul 17 19:09:09.287196 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:09.287129    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx" event={"ID":"c0271789-29c5-492c-b5fa-ee545a323fb5","Type":"ContainerDied","Data":"eedeb3fa85b504984d8d736ed94b3789d7048c1d594115f27e141d5191c93a41"}
Jul 17 19:09:09.287246 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:09.287200    2559 scope.go:117] "RemoveContainer" containerID="356561bd12afc3c1dab6c347e49d03ad3b47e5c284bc284c79feec04c18ed0c0"
Jul 17 19:09:09.287486 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:09.287465    2559 scope.go:117] "RemoveContainer" containerID="eedeb3fa85b504984d8d736ed94b3789d7048c1d594115f27e141d5191c93a41"
Jul 17 19:09:09.287681 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:09:09.287660    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx_odh-ai-gateway-infra(c0271789-29c5-492c-b5fa-ee545a323fb5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx" podUID="c0271789-29c5-492c-b5fa-ee545a323fb5"
Jul 17 19:09:10.330761 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:10.330722    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx"]
Jul 17 19:09:10.464038 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:10.464014    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx"
Jul 17 19:09:10.554561 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:10.554531    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0271789-29c5-492c-b5fa-ee545a323fb5-maas-api-service-ca\") pod \"c0271789-29c5-492c-b5fa-ee545a323fb5\" (UID: \"c0271789-29c5-492c-b5fa-ee545a323fb5\") "
Jul 17 19:09:10.554871 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:10.554851    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0271789-29c5-492c-b5fa-ee545a323fb5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c0271789-29c5-492c-b5fa-ee545a323fb5" (UID: "c0271789-29c5-492c-b5fa-ee545a323fb5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:09:10.655125 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:10.655098    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0271789-29c5-492c-b5fa-ee545a323fb5-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:09:11.301595 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:11.301570    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx"
Jul 17 19:09:11.301756 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:11.301613    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx" event={"ID":"c0271789-29c5-492c-b5fa-ee545a323fb5","Type":"ContainerDied","Data":"3daed345de1d127498a2bf8dd563bdff97f103ba520a57db461d35ebe9b5f86f"}
Jul 17 19:09:11.301756 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:11.301702    2559 scope.go:117] "RemoveContainer" containerID="eedeb3fa85b504984d8d736ed94b3789d7048c1d594115f27e141d5191c93a41"
Jul 17 19:09:11.322478 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:11.322452    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx"]
Jul 17 19:09:11.326601 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:11.326581    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-6f6lx"]
Jul 17 19:09:12.125903 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:12.125864    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0271789-29c5-492c-b5fa-ee545a323fb5" path="/var/lib/kubelet/pods/c0271789-29c5-492c-b5fa-ee545a323fb5/volumes"
Jul 17 19:09:22.120284 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:22.120251    2559 scope.go:117] "RemoveContainer" containerID="4f01a94b16f18968b364b07426096c79ead29e725da0e89fba568b736b0ae63d"
Jul 17 19:09:22.300568 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:22.300538    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj"]
Jul 17 19:09:22.300925 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:22.300913    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0271789-29c5-492c-b5fa-ee545a323fb5" containerName="revoke-keys"
Jul 17 19:09:22.300968 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:22.300927    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0271789-29c5-492c-b5fa-ee545a323fb5" containerName="revoke-keys"
Jul 17 19:09:22.300968 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:22.300940    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0271789-29c5-492c-b5fa-ee545a323fb5" containerName="revoke-keys"
Jul 17 19:09:22.300968 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:22.300945    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0271789-29c5-492c-b5fa-ee545a323fb5" containerName="revoke-keys"
Jul 17 19:09:22.301070 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:22.301009    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0271789-29c5-492c-b5fa-ee545a323fb5" containerName="revoke-keys"
Jul 17 19:09:22.301070 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:22.301021    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0271789-29c5-492c-b5fa-ee545a323fb5" containerName="revoke-keys"
Jul 17 19:09:22.301070 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:22.301029    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0271789-29c5-492c-b5fa-ee545a323fb5" containerName="revoke-keys"
Jul 17 19:09:22.305236 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:22.305221    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj"
Jul 17 19:09:22.311046 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:22.311022    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj"]
Jul 17 19:09:22.464871 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:22.464838    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f09804-488c-4008-808d-7f72daccc6a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj\" (UID: \"d1f09804-488c-4008-808d-7f72daccc6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj"
Jul 17 19:09:22.566145 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:22.566110    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f09804-488c-4008-808d-7f72daccc6a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj\" (UID: \"d1f09804-488c-4008-808d-7f72daccc6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj"
Jul 17 19:09:22.566723 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:22.566706    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f09804-488c-4008-808d-7f72daccc6a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj\" (UID: \"d1f09804-488c-4008-808d-7f72daccc6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj"
Jul 17 19:09:22.618135 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:22.618100    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj"
Jul 17 19:09:22.763121 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:22.763051    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj"]
Jul 17 19:09:22.765640 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:09:22.765614    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f09804_488c_4008_808d_7f72daccc6a9.slice/crio-dd2774a997df91367e3cf26e377d835cf788701269c6643fddb7154adc606a7a WatchSource:0}: Error finding container dd2774a997df91367e3cf26e377d835cf788701269c6643fddb7154adc606a7a: Status 404 returned error can't find the container with id dd2774a997df91367e3cf26e377d835cf788701269c6643fddb7154adc606a7a
Jul 17 19:09:23.373142 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:23.373101    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj" event={"ID":"d1f09804-488c-4008-808d-7f72daccc6a9","Type":"ContainerStarted","Data":"92a0422b2f4587ef2c7a3718413c96274576cf151205b3efa8dbb78da7896bfa"}
Jul 17 19:09:23.373553 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:23.373197    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj" event={"ID":"d1f09804-488c-4008-808d-7f72daccc6a9","Type":"ContainerStarted","Data":"dd2774a997df91367e3cf26e377d835cf788701269c6643fddb7154adc606a7a"}
Jul 17 19:09:23.375020 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:23.374995    2559 generic.go:356] "Generic (PLEG): container finished" podID="86b76a98-c448-4162-bb65-768d10f8c60b" containerID="c87a44f16e7880db9d0b24ce10126b9ab85dd090df0bd213eb46ed502d61c5c9" exitCode=6
Jul 17 19:09:23.375132 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:23.375060    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg" event={"ID":"86b76a98-c448-4162-bb65-768d10f8c60b","Type":"ContainerDied","Data":"c87a44f16e7880db9d0b24ce10126b9ab85dd090df0bd213eb46ed502d61c5c9"}
Jul 17 19:09:23.375132 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:23.375107    2559 scope.go:117] "RemoveContainer" containerID="4f01a94b16f18968b364b07426096c79ead29e725da0e89fba568b736b0ae63d"
Jul 17 19:09:23.375371 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:23.375356    2559 scope.go:117] "RemoveContainer" containerID="c87a44f16e7880db9d0b24ce10126b9ab85dd090df0bd213eb46ed502d61c5c9"
Jul 17 19:09:23.375564 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:09:23.375547    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg_odh-ai-gateway-infra(86b76a98-c448-4162-bb65-768d10f8c60b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg" podUID="86b76a98-c448-4162-bb65-768d10f8c60b"
Jul 17 19:09:23.389535 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:23.389500    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj" podStartSLOduration=1.389488431 podStartE2EDuration="1.389488431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:09:23.388184114 +0000 UTC m=+2173.829692806" watchObservedRunningTime="2026-07-17 19:09:23.389488431 +0000 UTC m=+2173.830997120"
Jul 17 19:09:24.384256 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:24.384222    2559 generic.go:356] "Generic (PLEG): container finished" podID="d1f09804-488c-4008-808d-7f72daccc6a9" containerID="92a0422b2f4587ef2c7a3718413c96274576cf151205b3efa8dbb78da7896bfa" exitCode=7
Jul 17 19:09:24.384796 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:24.384331    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj" event={"ID":"d1f09804-488c-4008-808d-7f72daccc6a9","Type":"ContainerDied","Data":"92a0422b2f4587ef2c7a3718413c96274576cf151205b3efa8dbb78da7896bfa"}
Jul 17 19:09:24.384796 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:24.384537    2559 scope.go:117] "RemoveContainer" containerID="92a0422b2f4587ef2c7a3718413c96274576cf151205b3efa8dbb78da7896bfa"
Jul 17 19:09:24.422787 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:24.422759    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg"]
Jul 17 19:09:24.565884 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:24.565859    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg"
Jul 17 19:09:24.685758 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:24.685717    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86b76a98-c448-4162-bb65-768d10f8c60b-maas-api-service-ca\") pod \"86b76a98-c448-4162-bb65-768d10f8c60b\" (UID: \"86b76a98-c448-4162-bb65-768d10f8c60b\") "
Jul 17 19:09:24.686101 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:24.686076    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b76a98-c448-4162-bb65-768d10f8c60b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86b76a98-c448-4162-bb65-768d10f8c60b" (UID: "86b76a98-c448-4162-bb65-768d10f8c60b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:09:24.786904 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:24.786870    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86b76a98-c448-4162-bb65-768d10f8c60b-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:09:25.392062 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:25.392030    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg"
Jul 17 19:09:25.392551 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:25.392063    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg" event={"ID":"86b76a98-c448-4162-bb65-768d10f8c60b","Type":"ContainerDied","Data":"b04363b4cc75a451885aa61670cb0d9014da67147d250e71bfe383b7f63c345a"}
Jul 17 19:09:25.392551 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:25.392131    2559 scope.go:117] "RemoveContainer" containerID="c87a44f16e7880db9d0b24ce10126b9ab85dd090df0bd213eb46ed502d61c5c9"
Jul 17 19:09:25.394262 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:25.394235    2559 generic.go:356] "Generic (PLEG): container finished" podID="d1f09804-488c-4008-808d-7f72daccc6a9" containerID="488bdfb8d4d4028c036e2173ccb9bbaab22a7d420eb5be6106137c589321a20a" exitCode=7
Jul 17 19:09:25.394426 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:25.394274    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj" event={"ID":"d1f09804-488c-4008-808d-7f72daccc6a9","Type":"ContainerDied","Data":"488bdfb8d4d4028c036e2173ccb9bbaab22a7d420eb5be6106137c589321a20a"}
Jul 17 19:09:25.394590 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:25.394572    2559 scope.go:117] "RemoveContainer" containerID="488bdfb8d4d4028c036e2173ccb9bbaab22a7d420eb5be6106137c589321a20a"
Jul 17 19:09:25.394818 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:09:25.394795    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj_odh-ai-gateway-infra(d1f09804-488c-4008-808d-7f72daccc6a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj" podUID="d1f09804-488c-4008-808d-7f72daccc6a9"
Jul 17 19:09:25.404507 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:25.404444    2559 scope.go:117] "RemoveContainer" containerID="92a0422b2f4587ef2c7a3718413c96274576cf151205b3efa8dbb78da7896bfa"
Jul 17 19:09:25.426798 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:25.426768    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg"]
Jul 17 19:09:25.431251 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:25.431228    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2lhsg"]
Jul 17 19:09:26.125194 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:26.125161    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86b76a98-c448-4162-bb65-768d10f8c60b" path="/var/lib/kubelet/pods/86b76a98-c448-4162-bb65-768d10f8c60b/volumes"
Jul 17 19:09:26.402046 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:26.401956    2559 scope.go:117] "RemoveContainer" containerID="488bdfb8d4d4028c036e2173ccb9bbaab22a7d420eb5be6106137c589321a20a"
Jul 17 19:09:26.402485 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:09:26.402161    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj_odh-ai-gateway-infra(d1f09804-488c-4008-808d-7f72daccc6a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj" podUID="d1f09804-488c-4008-808d-7f72daccc6a9"
Jul 17 19:09:39.119891 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:39.119856    2559 scope.go:117] "RemoveContainer" containerID="488bdfb8d4d4028c036e2173ccb9bbaab22a7d420eb5be6106137c589321a20a"
Jul 17 19:09:39.473894 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:39.473819    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj" event={"ID":"d1f09804-488c-4008-808d-7f72daccc6a9","Type":"ContainerStarted","Data":"ca9995adb67eea875611ddb0d49503c9d23e09cc9eb8318fab2143624fa89fd2"}
Jul 17 19:09:40.481170 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:40.481118    2559 generic.go:356] "Generic (PLEG): container finished" podID="d1f09804-488c-4008-808d-7f72daccc6a9" containerID="ca9995adb67eea875611ddb0d49503c9d23e09cc9eb8318fab2143624fa89fd2" exitCode=7
Jul 17 19:09:40.481649 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:40.481173    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj" event={"ID":"d1f09804-488c-4008-808d-7f72daccc6a9","Type":"ContainerDied","Data":"ca9995adb67eea875611ddb0d49503c9d23e09cc9eb8318fab2143624fa89fd2"}
Jul 17 19:09:40.481649 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:40.481230    2559 scope.go:117] "RemoveContainer" containerID="488bdfb8d4d4028c036e2173ccb9bbaab22a7d420eb5be6106137c589321a20a"
Jul 17 19:09:40.481649 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:40.481542    2559 scope.go:117] "RemoveContainer" containerID="ca9995adb67eea875611ddb0d49503c9d23e09cc9eb8318fab2143624fa89fd2"
Jul 17 19:09:40.481822 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:09:40.481783    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj_odh-ai-gateway-infra(d1f09804-488c-4008-808d-7f72daccc6a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj" podUID="d1f09804-488c-4008-808d-7f72daccc6a9"
Jul 17 19:09:40.504993 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:40.504967    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj"]
Jul 17 19:09:41.629143 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:41.629107    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj"
Jul 17 19:09:41.730457 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:41.730358    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f09804-488c-4008-808d-7f72daccc6a9-maas-api-service-ca\") pod \"d1f09804-488c-4008-808d-7f72daccc6a9\" (UID: \"d1f09804-488c-4008-808d-7f72daccc6a9\") "
Jul 17 19:09:41.730673 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:41.730652    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f09804-488c-4008-808d-7f72daccc6a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d1f09804-488c-4008-808d-7f72daccc6a9" (UID: "d1f09804-488c-4008-808d-7f72daccc6a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:09:41.730772 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:41.730760    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f09804-488c-4008-808d-7f72daccc6a9-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:09:42.495222 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:42.495142    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj"
Jul 17 19:09:42.495222 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:42.495183    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj" event={"ID":"d1f09804-488c-4008-808d-7f72daccc6a9","Type":"ContainerDied","Data":"dd2774a997df91367e3cf26e377d835cf788701269c6643fddb7154adc606a7a"}
Jul 17 19:09:42.495439 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:42.495248    2559 scope.go:117] "RemoveContainer" containerID="ca9995adb67eea875611ddb0d49503c9d23e09cc9eb8318fab2143624fa89fd2"
Jul 17 19:09:42.512415 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:42.512390    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj"]
Jul 17 19:09:42.517486 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:42.517465    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-9rbmj"]
Jul 17 19:09:43.463882 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.463847    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9"]
Jul 17 19:09:43.464265 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.464247    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86b76a98-c448-4162-bb65-768d10f8c60b" containerName="revoke-keys"
Jul 17 19:09:43.464265 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.464259    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="86b76a98-c448-4162-bb65-768d10f8c60b" containerName="revoke-keys"
Jul 17 19:09:43.464366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.464279    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0271789-29c5-492c-b5fa-ee545a323fb5" containerName="revoke-keys"
Jul 17 19:09:43.464366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.464288    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0271789-29c5-492c-b5fa-ee545a323fb5" containerName="revoke-keys"
Jul 17 19:09:43.464366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.464297    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86b76a98-c448-4162-bb65-768d10f8c60b" containerName="revoke-keys"
Jul 17 19:09:43.464366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.464316    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="86b76a98-c448-4162-bb65-768d10f8c60b" containerName="revoke-keys"
Jul 17 19:09:43.464366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.464324    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86b76a98-c448-4162-bb65-768d10f8c60b" containerName="revoke-keys"
Jul 17 19:09:43.464366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.464330    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="86b76a98-c448-4162-bb65-768d10f8c60b" containerName="revoke-keys"
Jul 17 19:09:43.464366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.464337    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1f09804-488c-4008-808d-7f72daccc6a9" containerName="revoke-keys"
Jul 17 19:09:43.464366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.464342    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1f09804-488c-4008-808d-7f72daccc6a9" containerName="revoke-keys"
Jul 17 19:09:43.464366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.464365    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1f09804-488c-4008-808d-7f72daccc6a9" containerName="revoke-keys"
Jul 17 19:09:43.464632 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.464374    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1f09804-488c-4008-808d-7f72daccc6a9" containerName="revoke-keys"
Jul 17 19:09:43.464632 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.464438    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1f09804-488c-4008-808d-7f72daccc6a9" containerName="revoke-keys"
Jul 17 19:09:43.464632 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.464451    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="86b76a98-c448-4162-bb65-768d10f8c60b" containerName="revoke-keys"
Jul 17 19:09:43.464632 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.464458    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1f09804-488c-4008-808d-7f72daccc6a9" containerName="revoke-keys"
Jul 17 19:09:43.464632 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.464463    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1f09804-488c-4008-808d-7f72daccc6a9" containerName="revoke-keys"
Jul 17 19:09:43.464632 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.464469    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="86b76a98-c448-4162-bb65-768d10f8c60b" containerName="revoke-keys"
Jul 17 19:09:43.464632 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.464476    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="86b76a98-c448-4162-bb65-768d10f8c60b" containerName="revoke-keys"
Jul 17 19:09:43.467497 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.467480    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9"
Jul 17 19:09:43.470077 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.470060    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:09:43.476073 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.476051    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9"]
Jul 17 19:09:43.547545 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.547512    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3925501d-62c1-43e6-8f75-1951667b33bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9\" (UID: \"3925501d-62c1-43e6-8f75-1951667b33bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9"
Jul 17 19:09:43.648949 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.648916    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3925501d-62c1-43e6-8f75-1951667b33bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9\" (UID: \"3925501d-62c1-43e6-8f75-1951667b33bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9"
Jul 17 19:09:43.649653 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.649628    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3925501d-62c1-43e6-8f75-1951667b33bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9\" (UID: \"3925501d-62c1-43e6-8f75-1951667b33bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9"
Jul 17 19:09:43.779868 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.779784    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9"
Jul 17 19:09:43.913778 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:43.913753    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9"]
Jul 17 19:09:43.915602 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:09:43.915578    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3925501d_62c1_43e6_8f75_1951667b33bb.slice/crio-2cc46b5ad416e086522b9d00eefcc79d4b5a83384103e466d201a41a7c5d5022 WatchSource:0}: Error finding container 2cc46b5ad416e086522b9d00eefcc79d4b5a83384103e466d201a41a7c5d5022: Status 404 returned error can't find the container with id 2cc46b5ad416e086522b9d00eefcc79d4b5a83384103e466d201a41a7c5d5022
Jul 17 19:09:44.126199 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:44.126169    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1f09804-488c-4008-808d-7f72daccc6a9" path="/var/lib/kubelet/pods/d1f09804-488c-4008-808d-7f72daccc6a9/volumes"
Jul 17 19:09:44.513050 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:44.512962    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9" event={"ID":"3925501d-62c1-43e6-8f75-1951667b33bb","Type":"ContainerStarted","Data":"1b0cacaced7d2d5cb05b035e2c01b73f8fff400fb8fbf81b5fc11516a5929a7a"}
Jul 17 19:09:44.513050 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:44.513025    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9" event={"ID":"3925501d-62c1-43e6-8f75-1951667b33bb","Type":"ContainerStarted","Data":"2cc46b5ad416e086522b9d00eefcc79d4b5a83384103e466d201a41a7c5d5022"}
Jul 17 19:09:44.531925 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:44.531872    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9" podStartSLOduration=1.531856464 podStartE2EDuration="1.531856464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:09:44.528940819 +0000 UTC m=+2194.970449506" watchObservedRunningTime="2026-07-17 19:09:44.531856464 +0000 UTC m=+2194.973365153"
Jul 17 19:09:45.520031 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:45.519995    2559 generic.go:356] "Generic (PLEG): container finished" podID="3925501d-62c1-43e6-8f75-1951667b33bb" containerID="1b0cacaced7d2d5cb05b035e2c01b73f8fff400fb8fbf81b5fc11516a5929a7a" exitCode=6
Jul 17 19:09:45.520509 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:45.520078    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9" event={"ID":"3925501d-62c1-43e6-8f75-1951667b33bb","Type":"ContainerDied","Data":"1b0cacaced7d2d5cb05b035e2c01b73f8fff400fb8fbf81b5fc11516a5929a7a"}
Jul 17 19:09:45.520509 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:45.520193    2559 scope.go:117] "RemoveContainer" containerID="1b0cacaced7d2d5cb05b035e2c01b73f8fff400fb8fbf81b5fc11516a5929a7a"
Jul 17 19:09:46.528113 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:46.528019    2559 generic.go:356] "Generic (PLEG): container finished" podID="3925501d-62c1-43e6-8f75-1951667b33bb" containerID="47f128ab4e29b18d6073d3be2f67fbc045083638a3bbe41cc9d43956550e261f" exitCode=6
Jul 17 19:09:46.528586 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:46.528118    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9" event={"ID":"3925501d-62c1-43e6-8f75-1951667b33bb","Type":"ContainerDied","Data":"47f128ab4e29b18d6073d3be2f67fbc045083638a3bbe41cc9d43956550e261f"}
Jul 17 19:09:46.528586 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:46.528185    2559 scope.go:117] "RemoveContainer" containerID="1b0cacaced7d2d5cb05b035e2c01b73f8fff400fb8fbf81b5fc11516a5929a7a"
Jul 17 19:09:46.528586 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:46.528352    2559 scope.go:117] "RemoveContainer" containerID="47f128ab4e29b18d6073d3be2f67fbc045083638a3bbe41cc9d43956550e261f"
Jul 17 19:09:46.528715 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:09:46.528617    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9_odh-ai-gateway-infra(3925501d-62c1-43e6-8f75-1951667b33bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9" podUID="3925501d-62c1-43e6-8f75-1951667b33bb"
Jul 17 19:09:47.535494 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:09:47.535461    2559 scope.go:117] "RemoveContainer" containerID="47f128ab4e29b18d6073d3be2f67fbc045083638a3bbe41cc9d43956550e261f"
Jul 17 19:09:47.535873 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:09:47.535657    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9_odh-ai-gateway-infra(3925501d-62c1-43e6-8f75-1951667b33bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9" podUID="3925501d-62c1-43e6-8f75-1951667b33bb"
Jul 17 19:10:01.120188 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:01.120159    2559 scope.go:117] "RemoveContainer" containerID="47f128ab4e29b18d6073d3be2f67fbc045083638a3bbe41cc9d43956550e261f"
Jul 17 19:10:01.615593 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:01.615550    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9" event={"ID":"3925501d-62c1-43e6-8f75-1951667b33bb","Type":"ContainerStarted","Data":"75908ae9888b60db4637a2a6495d3b12d759bf283c8e4e6d93b18aa53b1ef698"}
Jul 17 19:10:02.622812 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:02.622778    2559 generic.go:356] "Generic (PLEG): container finished" podID="3925501d-62c1-43e6-8f75-1951667b33bb" containerID="75908ae9888b60db4637a2a6495d3b12d759bf283c8e4e6d93b18aa53b1ef698" exitCode=6
Jul 17 19:10:02.623230 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:02.622819    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9" event={"ID":"3925501d-62c1-43e6-8f75-1951667b33bb","Type":"ContainerDied","Data":"75908ae9888b60db4637a2a6495d3b12d759bf283c8e4e6d93b18aa53b1ef698"}
Jul 17 19:10:02.623230 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:02.622875    2559 scope.go:117] "RemoveContainer" containerID="47f128ab4e29b18d6073d3be2f67fbc045083638a3bbe41cc9d43956550e261f"
Jul 17 19:10:02.623230 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:02.623141    2559 scope.go:117] "RemoveContainer" containerID="75908ae9888b60db4637a2a6495d3b12d759bf283c8e4e6d93b18aa53b1ef698"
Jul 17 19:10:02.623393 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:10:02.623373    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9_odh-ai-gateway-infra(3925501d-62c1-43e6-8f75-1951667b33bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9" podUID="3925501d-62c1-43e6-8f75-1951667b33bb"
Jul 17 19:10:02.650437 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:02.650408    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9"]
Jul 17 19:10:03.759320 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:03.759279    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9"
Jul 17 19:10:03.831108 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:03.831078    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3925501d-62c1-43e6-8f75-1951667b33bb-maas-api-service-ca\") pod \"3925501d-62c1-43e6-8f75-1951667b33bb\" (UID: \"3925501d-62c1-43e6-8f75-1951667b33bb\") "
Jul 17 19:10:03.831498 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:03.831471    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3925501d-62c1-43e6-8f75-1951667b33bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3925501d-62c1-43e6-8f75-1951667b33bb" (UID: "3925501d-62c1-43e6-8f75-1951667b33bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:10:03.932428 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:03.932346    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3925501d-62c1-43e6-8f75-1951667b33bb-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:10:04.638449 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:04.638420    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9"
Jul 17 19:10:04.638449 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:04.638449    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9" event={"ID":"3925501d-62c1-43e6-8f75-1951667b33bb","Type":"ContainerDied","Data":"2cc46b5ad416e086522b9d00eefcc79d4b5a83384103e466d201a41a7c5d5022"}
Jul 17 19:10:04.638733 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:04.638517    2559 scope.go:117] "RemoveContainer" containerID="75908ae9888b60db4637a2a6495d3b12d759bf283c8e4e6d93b18aa53b1ef698"
Jul 17 19:10:04.660233 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:04.660203    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9"]
Jul 17 19:10:04.665869 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:04.665848    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-2t2l9"]
Jul 17 19:10:05.285890 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:05.285859    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh"]
Jul 17 19:10:05.286271 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:05.286224    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3925501d-62c1-43e6-8f75-1951667b33bb" containerName="revoke-keys"
Jul 17 19:10:05.286271 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:05.286234    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3925501d-62c1-43e6-8f75-1951667b33bb" containerName="revoke-keys"
Jul 17 19:10:05.286271 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:05.286245    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3925501d-62c1-43e6-8f75-1951667b33bb" containerName="revoke-keys"
Jul 17 19:10:05.286271 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:05.286251    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3925501d-62c1-43e6-8f75-1951667b33bb" containerName="revoke-keys"
Jul 17 19:10:05.286271 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:05.286256    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3925501d-62c1-43e6-8f75-1951667b33bb" containerName="revoke-keys"
Jul 17 19:10:05.286271 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:05.286261    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3925501d-62c1-43e6-8f75-1951667b33bb" containerName="revoke-keys"
Jul 17 19:10:05.286271 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:05.286269    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1f09804-488c-4008-808d-7f72daccc6a9" containerName="revoke-keys"
Jul 17 19:10:05.286271 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:05.286275    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1f09804-488c-4008-808d-7f72daccc6a9" containerName="revoke-keys"
Jul 17 19:10:05.286555 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:05.286362    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3925501d-62c1-43e6-8f75-1951667b33bb" containerName="revoke-keys"
Jul 17 19:10:05.286555 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:05.286375    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3925501d-62c1-43e6-8f75-1951667b33bb" containerName="revoke-keys"
Jul 17 19:10:05.290559 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:05.290543    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh"
Jul 17 19:10:05.293394 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:05.293374    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:10:05.298238 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:05.298187    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh"]
Jul 17 19:10:05.444776 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:05.444732    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68f6068-a393-4063-b40f-67692134eba5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh\" (UID: \"f68f6068-a393-4063-b40f-67692134eba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh"
Jul 17 19:10:05.545375 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:05.545282    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68f6068-a393-4063-b40f-67692134eba5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh\" (UID: \"f68f6068-a393-4063-b40f-67692134eba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh"
Jul 17 19:10:05.545953 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:05.545925    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68f6068-a393-4063-b40f-67692134eba5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh\" (UID: \"f68f6068-a393-4063-b40f-67692134eba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh"
Jul 17 19:10:05.603357 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:05.603290    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh"
Jul 17 19:10:05.738198 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:05.738169    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh"]
Jul 17 19:10:05.739643 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:10:05.739617    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68f6068_a393_4063_b40f_67692134eba5.slice/crio-380e7dc3cd5031945a22aaed85bb524ded8aee770562d54fec4136671abc3b89 WatchSource:0}: Error finding container 380e7dc3cd5031945a22aaed85bb524ded8aee770562d54fec4136671abc3b89: Status 404 returned error can't find the container with id 380e7dc3cd5031945a22aaed85bb524ded8aee770562d54fec4136671abc3b89
Jul 17 19:10:06.125820 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:06.125745    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3925501d-62c1-43e6-8f75-1951667b33bb" path="/var/lib/kubelet/pods/3925501d-62c1-43e6-8f75-1951667b33bb/volumes"
Jul 17 19:10:06.651935 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:06.651904    2559 generic.go:356] "Generic (PLEG): container finished" podID="f68f6068-a393-4063-b40f-67692134eba5" containerID="e34770964590aa2c7961eecc60bf638e1f34e58ea72c51d0f8a3db0ac3bcdc19" exitCode=6
Jul 17 19:10:06.652457 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:06.651933    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh" event={"ID":"f68f6068-a393-4063-b40f-67692134eba5","Type":"ContainerDied","Data":"e34770964590aa2c7961eecc60bf638e1f34e58ea72c51d0f8a3db0ac3bcdc19"}
Jul 17 19:10:06.652457 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:06.652032    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh" event={"ID":"f68f6068-a393-4063-b40f-67692134eba5","Type":"ContainerStarted","Data":"380e7dc3cd5031945a22aaed85bb524ded8aee770562d54fec4136671abc3b89"}
Jul 17 19:10:06.652457 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:06.652285    2559 scope.go:117] "RemoveContainer" containerID="e34770964590aa2c7961eecc60bf638e1f34e58ea72c51d0f8a3db0ac3bcdc19"
Jul 17 19:10:07.659482 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:07.659451    2559 generic.go:356] "Generic (PLEG): container finished" podID="f68f6068-a393-4063-b40f-67692134eba5" containerID="bd6acda636e3c21ca904cef55fb4ffdfa90b1f0a7a79dfd98b26c32bc901f3a5" exitCode=6
Jul 17 19:10:07.659912 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:07.659537    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh" event={"ID":"f68f6068-a393-4063-b40f-67692134eba5","Type":"ContainerDied","Data":"bd6acda636e3c21ca904cef55fb4ffdfa90b1f0a7a79dfd98b26c32bc901f3a5"}
Jul 17 19:10:07.659912 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:07.659603    2559 scope.go:117] "RemoveContainer" containerID="e34770964590aa2c7961eecc60bf638e1f34e58ea72c51d0f8a3db0ac3bcdc19"
Jul 17 19:10:07.659912 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:07.659739    2559 scope.go:117] "RemoveContainer" containerID="bd6acda636e3c21ca904cef55fb4ffdfa90b1f0a7a79dfd98b26c32bc901f3a5"
Jul 17 19:10:07.660034 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:10:07.659952    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh_odh-ai-gateway-infra(f68f6068-a393-4063-b40f-67692134eba5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh" podUID="f68f6068-a393-4063-b40f-67692134eba5"
Jul 17 19:10:08.667376 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:08.667350    2559 scope.go:117] "RemoveContainer" containerID="bd6acda636e3c21ca904cef55fb4ffdfa90b1f0a7a79dfd98b26c32bc901f3a5"
Jul 17 19:10:08.667757 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:10:08.667559    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh_odh-ai-gateway-infra(f68f6068-a393-4063-b40f-67692134eba5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh" podUID="f68f6068-a393-4063-b40f-67692134eba5"
Jul 17 19:10:21.120044 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:21.120017    2559 scope.go:117] "RemoveContainer" containerID="bd6acda636e3c21ca904cef55fb4ffdfa90b1f0a7a79dfd98b26c32bc901f3a5"
Jul 17 19:10:21.740161 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:21.740094    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh" event={"ID":"f68f6068-a393-4063-b40f-67692134eba5","Type":"ContainerStarted","Data":"42a80d9391daef9b027ce60f27e9ccc81cdf2dcba0b822a3b5041880fb62c644"}
Jul 17 19:10:21.755761 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:21.755717    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh" podStartSLOduration=16.755698458 podStartE2EDuration="16.755698458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:10:21.755237438 +0000 UTC m=+2232.196746134" watchObservedRunningTime="2026-07-17 19:10:21.755698458 +0000 UTC m=+2232.197207157"
Jul 17 19:10:22.348414 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:22.348380    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb"]
Jul 17 19:10:22.348869 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:22.348856    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3925501d-62c1-43e6-8f75-1951667b33bb" containerName="revoke-keys"
Jul 17 19:10:22.352026 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:22.352009    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb"
Jul 17 19:10:22.361216 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:22.361188    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb"]
Jul 17 19:10:22.396965 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:22.396943    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7d066c-d312-472d-b005-51a21e772cd8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb\" (UID: \"0a7d066c-d312-472d-b005-51a21e772cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb"
Jul 17 19:10:22.497737 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:22.497704    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7d066c-d312-472d-b005-51a21e772cd8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb\" (UID: \"0a7d066c-d312-472d-b005-51a21e772cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb"
Jul 17 19:10:22.498300 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:22.498282    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7d066c-d312-472d-b005-51a21e772cd8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb\" (UID: \"0a7d066c-d312-472d-b005-51a21e772cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb"
Jul 17 19:10:22.663445 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:22.663385    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb"
Jul 17 19:10:22.748065 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:22.748035    2559 generic.go:356] "Generic (PLEG): container finished" podID="f68f6068-a393-4063-b40f-67692134eba5" containerID="42a80d9391daef9b027ce60f27e9ccc81cdf2dcba0b822a3b5041880fb62c644" exitCode=6
Jul 17 19:10:22.748197 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:22.748100    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh" event={"ID":"f68f6068-a393-4063-b40f-67692134eba5","Type":"ContainerDied","Data":"42a80d9391daef9b027ce60f27e9ccc81cdf2dcba0b822a3b5041880fb62c644"}
Jul 17 19:10:22.748197 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:22.748159    2559 scope.go:117] "RemoveContainer" containerID="bd6acda636e3c21ca904cef55fb4ffdfa90b1f0a7a79dfd98b26c32bc901f3a5"
Jul 17 19:10:22.748484 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:22.748467    2559 scope.go:117] "RemoveContainer" containerID="42a80d9391daef9b027ce60f27e9ccc81cdf2dcba0b822a3b5041880fb62c644"
Jul 17 19:10:22.748713 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:10:22.748692    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh_odh-ai-gateway-infra(f68f6068-a393-4063-b40f-67692134eba5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh" podUID="f68f6068-a393-4063-b40f-67692134eba5"
Jul 17 19:10:22.771636 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:22.771611    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh"]
Jul 17 19:10:22.818910 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:22.818890    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb"]
Jul 17 19:10:22.820516 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:10:22.820493    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7d066c_d312_472d_b005_51a21e772cd8.slice/crio-1baaeb922dfd9741e97002c8202958b96e4fa9b18aadc28dbdfda72522efa161 WatchSource:0}: Error finding container 1baaeb922dfd9741e97002c8202958b96e4fa9b18aadc28dbdfda72522efa161: Status 404 returned error can't find the container with id 1baaeb922dfd9741e97002c8202958b96e4fa9b18aadc28dbdfda72522efa161
Jul 17 19:10:23.756370 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:23.756258    2559 generic.go:356] "Generic (PLEG): container finished" podID="0a7d066c-d312-472d-b005-51a21e772cd8" containerID="2603ede5068d7cb5d43c6c47017bbacda0c2be332c6e102f6fbc94adb0ca2636" exitCode=7
Jul 17 19:10:23.756370 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:23.756352    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb" event={"ID":"0a7d066c-d312-472d-b005-51a21e772cd8","Type":"ContainerDied","Data":"2603ede5068d7cb5d43c6c47017bbacda0c2be332c6e102f6fbc94adb0ca2636"}
Jul 17 19:10:23.756836 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:23.756447    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb" event={"ID":"0a7d066c-d312-472d-b005-51a21e772cd8","Type":"ContainerStarted","Data":"1baaeb922dfd9741e97002c8202958b96e4fa9b18aadc28dbdfda72522efa161"}
Jul 17 19:10:23.756836 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:23.756553    2559 scope.go:117] "RemoveContainer" containerID="2603ede5068d7cb5d43c6c47017bbacda0c2be332c6e102f6fbc94adb0ca2636"
Jul 17 19:10:23.913323 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:23.913289    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh"
Jul 17 19:10:24.010510 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:24.010447    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68f6068-a393-4063-b40f-67692134eba5-maas-api-service-ca\") pod \"f68f6068-a393-4063-b40f-67692134eba5\" (UID: \"f68f6068-a393-4063-b40f-67692134eba5\") "
Jul 17 19:10:24.010807 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:24.010787    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f68f6068-a393-4063-b40f-67692134eba5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f68f6068-a393-4063-b40f-67692134eba5" (UID: "f68f6068-a393-4063-b40f-67692134eba5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:10:24.111832 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:24.111802    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f68f6068-a393-4063-b40f-67692134eba5-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:10:24.765452 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:24.765368    2559 generic.go:356] "Generic (PLEG): container finished" podID="0a7d066c-d312-472d-b005-51a21e772cd8" containerID="91dcc22d228fe48d752b8c12d987b43ba2d60f55cd72a99cf58bfe40f66ae874" exitCode=7
Jul 17 19:10:24.765929 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:24.765466    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb" event={"ID":"0a7d066c-d312-472d-b005-51a21e772cd8","Type":"ContainerDied","Data":"91dcc22d228fe48d752b8c12d987b43ba2d60f55cd72a99cf58bfe40f66ae874"}
Jul 17 19:10:24.765929 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:24.765556    2559 scope.go:117] "RemoveContainer" containerID="2603ede5068d7cb5d43c6c47017bbacda0c2be332c6e102f6fbc94adb0ca2636"
Jul 17 19:10:24.765929 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:24.765714    2559 scope.go:117] "RemoveContainer" containerID="91dcc22d228fe48d752b8c12d987b43ba2d60f55cd72a99cf58bfe40f66ae874"
Jul 17 19:10:24.766075 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:10:24.765939    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb_odh-ai-gateway-infra(0a7d066c-d312-472d-b005-51a21e772cd8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb" podUID="0a7d066c-d312-472d-b005-51a21e772cd8"
Jul 17 19:10:24.767064 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:24.766907    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh" event={"ID":"f68f6068-a393-4063-b40f-67692134eba5","Type":"ContainerDied","Data":"380e7dc3cd5031945a22aaed85bb524ded8aee770562d54fec4136671abc3b89"}
Jul 17 19:10:24.767064 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:24.766957    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh"
Jul 17 19:10:24.778047 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:24.778024    2559 scope.go:117] "RemoveContainer" containerID="42a80d9391daef9b027ce60f27e9ccc81cdf2dcba0b822a3b5041880fb62c644"
Jul 17 19:10:24.796423 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:24.796400    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh"]
Jul 17 19:10:24.803573 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:24.803551    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-44zhh"]
Jul 17 19:10:25.774044 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:25.774014    2559 scope.go:117] "RemoveContainer" containerID="91dcc22d228fe48d752b8c12d987b43ba2d60f55cd72a99cf58bfe40f66ae874"
Jul 17 19:10:25.774511 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:10:25.774234    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb_odh-ai-gateway-infra(0a7d066c-d312-472d-b005-51a21e772cd8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb" podUID="0a7d066c-d312-472d-b005-51a21e772cd8"
Jul 17 19:10:26.125162 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:26.125131    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f68f6068-a393-4063-b40f-67692134eba5" path="/var/lib/kubelet/pods/f68f6068-a393-4063-b40f-67692134eba5/volumes"
Jul 17 19:10:40.123343 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:40.123255    2559 scope.go:117] "RemoveContainer" containerID="91dcc22d228fe48d752b8c12d987b43ba2d60f55cd72a99cf58bfe40f66ae874"
Jul 17 19:10:40.856447 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:40.856421    2559 generic.go:356] "Generic (PLEG): container finished" podID="0a7d066c-d312-472d-b005-51a21e772cd8" containerID="bb5782344a3f6c59fda0b4af8703c1c683383900816ad38f0500bad2ef61afd4" exitCode=7
Jul 17 19:10:40.856582 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:40.856467    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb" event={"ID":"0a7d066c-d312-472d-b005-51a21e772cd8","Type":"ContainerDied","Data":"bb5782344a3f6c59fda0b4af8703c1c683383900816ad38f0500bad2ef61afd4"}
Jul 17 19:10:40.856582 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:40.856535    2559 scope.go:117] "RemoveContainer" containerID="91dcc22d228fe48d752b8c12d987b43ba2d60f55cd72a99cf58bfe40f66ae874"
Jul 17 19:10:40.856818 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:40.856803    2559 scope.go:117] "RemoveContainer" containerID="bb5782344a3f6c59fda0b4af8703c1c683383900816ad38f0500bad2ef61afd4"
Jul 17 19:10:40.857008 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:10:40.856986    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb_odh-ai-gateway-infra(0a7d066c-d312-472d-b005-51a21e772cd8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb" podUID="0a7d066c-d312-472d-b005-51a21e772cd8"
Jul 17 19:10:41.885221 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:41.885191    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb"]
Jul 17 19:10:42.019940 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:42.019915    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb"
Jul 17 19:10:42.057821 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:42.057798    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7d066c-d312-472d-b005-51a21e772cd8-maas-api-service-ca\") pod \"0a7d066c-d312-472d-b005-51a21e772cd8\" (UID: \"0a7d066c-d312-472d-b005-51a21e772cd8\") "
Jul 17 19:10:42.058166 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:42.058145    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a7d066c-d312-472d-b005-51a21e772cd8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a7d066c-d312-472d-b005-51a21e772cd8" (UID: "0a7d066c-d312-472d-b005-51a21e772cd8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:10:42.159122 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:42.159089    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7d066c-d312-472d-b005-51a21e772cd8-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:10:42.871369 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:42.871332    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb" event={"ID":"0a7d066c-d312-472d-b005-51a21e772cd8","Type":"ContainerDied","Data":"1baaeb922dfd9741e97002c8202958b96e4fa9b18aadc28dbdfda72522efa161"}
Jul 17 19:10:42.871543 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:42.871373    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb"
Jul 17 19:10:42.871543 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:42.871393    2559 scope.go:117] "RemoveContainer" containerID="bb5782344a3f6c59fda0b4af8703c1c683383900816ad38f0500bad2ef61afd4"
Jul 17 19:10:42.889477 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:42.889444    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb"]
Jul 17 19:10:42.898369 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:42.898346    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kg9pb"]
Jul 17 19:10:43.511794 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.511764    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd"]
Jul 17 19:10:43.512134 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.512122    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f68f6068-a393-4063-b40f-67692134eba5" containerName="revoke-keys"
Jul 17 19:10:43.512179 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.512136    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f68f6068-a393-4063-b40f-67692134eba5" containerName="revoke-keys"
Jul 17 19:10:43.512179 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.512146    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a7d066c-d312-472d-b005-51a21e772cd8" containerName="revoke-keys"
Jul 17 19:10:43.512179 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.512151    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a7d066c-d312-472d-b005-51a21e772cd8" containerName="revoke-keys"
Jul 17 19:10:43.512179 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.512161    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f68f6068-a393-4063-b40f-67692134eba5" containerName="revoke-keys"
Jul 17 19:10:43.512179 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.512167    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f68f6068-a393-4063-b40f-67692134eba5" containerName="revoke-keys"
Jul 17 19:10:43.512179 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.512175    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a7d066c-d312-472d-b005-51a21e772cd8" containerName="revoke-keys"
Jul 17 19:10:43.512179 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.512181    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a7d066c-d312-472d-b005-51a21e772cd8" containerName="revoke-keys"
Jul 17 19:10:43.512447 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.512242    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f68f6068-a393-4063-b40f-67692134eba5" containerName="revoke-keys"
Jul 17 19:10:43.512447 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.512250    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a7d066c-d312-472d-b005-51a21e772cd8" containerName="revoke-keys"
Jul 17 19:10:43.512447 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.512256    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f68f6068-a393-4063-b40f-67692134eba5" containerName="revoke-keys"
Jul 17 19:10:43.512447 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.512263    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a7d066c-d312-472d-b005-51a21e772cd8" containerName="revoke-keys"
Jul 17 19:10:43.512447 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.512269    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a7d066c-d312-472d-b005-51a21e772cd8" containerName="revoke-keys"
Jul 17 19:10:43.515195 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.515180    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd"
Jul 17 19:10:43.517762 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.517740    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:10:43.522985 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.522614    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd"]
Jul 17 19:10:43.571725 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.571701    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/277174ba-8702-48d0-a3b3-cd8d57758263-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd\" (UID: \"277174ba-8702-48d0-a3b3-cd8d57758263\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd"
Jul 17 19:10:43.672687 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.672666    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/277174ba-8702-48d0-a3b3-cd8d57758263-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd\" (UID: \"277174ba-8702-48d0-a3b3-cd8d57758263\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd"
Jul 17 19:10:43.673270 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.673250    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/277174ba-8702-48d0-a3b3-cd8d57758263-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd\" (UID: \"277174ba-8702-48d0-a3b3-cd8d57758263\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd"
Jul 17 19:10:43.826716 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.826688    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd"
Jul 17 19:10:43.973156 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:10:43.973128    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277174ba_8702_48d0_a3b3_cd8d57758263.slice/crio-f014c924854db95a1787082a011368a9e8217fe18d49b38d3878137e39175cbb WatchSource:0}: Error finding container f014c924854db95a1787082a011368a9e8217fe18d49b38d3878137e39175cbb: Status 404 returned error can't find the container with id f014c924854db95a1787082a011368a9e8217fe18d49b38d3878137e39175cbb
Jul 17 19:10:43.973905 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:43.973880    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd"]
Jul 17 19:10:44.126106 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:44.126045    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a7d066c-d312-472d-b005-51a21e772cd8" path="/var/lib/kubelet/pods/0a7d066c-d312-472d-b005-51a21e772cd8/volumes"
Jul 17 19:10:44.886322 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:44.886232    2559 generic.go:356] "Generic (PLEG): container finished" podID="277174ba-8702-48d0-a3b3-cd8d57758263" containerID="4d9cced5d0f8b8098a1e53b0400c497c49943fc93fb8bcb4df8cefa2ffeeabe8" exitCode=6
Jul 17 19:10:44.886442 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:44.886324    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd" event={"ID":"277174ba-8702-48d0-a3b3-cd8d57758263","Type":"ContainerDied","Data":"4d9cced5d0f8b8098a1e53b0400c497c49943fc93fb8bcb4df8cefa2ffeeabe8"}
Jul 17 19:10:44.886442 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:44.886376    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd" event={"ID":"277174ba-8702-48d0-a3b3-cd8d57758263","Type":"ContainerStarted","Data":"f014c924854db95a1787082a011368a9e8217fe18d49b38d3878137e39175cbb"}
Jul 17 19:10:44.886528 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:44.886501    2559 scope.go:117] "RemoveContainer" containerID="4d9cced5d0f8b8098a1e53b0400c497c49943fc93fb8bcb4df8cefa2ffeeabe8"
Jul 17 19:10:45.894343 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:45.894292    2559 generic.go:356] "Generic (PLEG): container finished" podID="277174ba-8702-48d0-a3b3-cd8d57758263" containerID="99e8d7ccca329c58d521cef8cf78f5171bdf86f121b74ea476f73ab9efee967e" exitCode=6
Jul 17 19:10:45.894757 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:45.894352    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd" event={"ID":"277174ba-8702-48d0-a3b3-cd8d57758263","Type":"ContainerDied","Data":"99e8d7ccca329c58d521cef8cf78f5171bdf86f121b74ea476f73ab9efee967e"}
Jul 17 19:10:45.894757 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:45.894417    2559 scope.go:117] "RemoveContainer" containerID="4d9cced5d0f8b8098a1e53b0400c497c49943fc93fb8bcb4df8cefa2ffeeabe8"
Jul 17 19:10:45.894757 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:45.894571    2559 scope.go:117] "RemoveContainer" containerID="99e8d7ccca329c58d521cef8cf78f5171bdf86f121b74ea476f73ab9efee967e"
Jul 17 19:10:45.894882 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:10:45.894764    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd_odh-ai-gateway-infra(277174ba-8702-48d0-a3b3-cd8d57758263)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd" podUID="277174ba-8702-48d0-a3b3-cd8d57758263"
Jul 17 19:10:46.902472 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:46.902444    2559 scope.go:117] "RemoveContainer" containerID="99e8d7ccca329c58d521cef8cf78f5171bdf86f121b74ea476f73ab9efee967e"
Jul 17 19:10:46.902879 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:10:46.902604    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd_odh-ai-gateway-infra(277174ba-8702-48d0-a3b3-cd8d57758263)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd" podUID="277174ba-8702-48d0-a3b3-cd8d57758263"
Jul 17 19:10:55.332450 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:55.332411    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25"]
Jul 17 19:10:55.332849 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:55.332779    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a7d066c-d312-472d-b005-51a21e772cd8" containerName="revoke-keys"
Jul 17 19:10:55.332849 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:55.332793    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a7d066c-d312-472d-b005-51a21e772cd8" containerName="revoke-keys"
Jul 17 19:10:55.332849 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:55.332807    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f68f6068-a393-4063-b40f-67692134eba5" containerName="revoke-keys"
Jul 17 19:10:55.332849 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:55.332813    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f68f6068-a393-4063-b40f-67692134eba5" containerName="revoke-keys"
Jul 17 19:10:55.333053 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:55.332892    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f68f6068-a393-4063-b40f-67692134eba5" containerName="revoke-keys"
Jul 17 19:10:55.337079 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:55.337064    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25"
Jul 17 19:10:55.343833 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:55.343807    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25"]
Jul 17 19:10:55.462640 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:55.462608    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ffbceba-acda-4ab0-be47-623f932180a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25\" (UID: \"8ffbceba-acda-4ab0-be47-623f932180a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25"
Jul 17 19:10:55.563903 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:55.563877    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ffbceba-acda-4ab0-be47-623f932180a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25\" (UID: \"8ffbceba-acda-4ab0-be47-623f932180a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25"
Jul 17 19:10:55.564434 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:55.564417    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ffbceba-acda-4ab0-be47-623f932180a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25\" (UID: \"8ffbceba-acda-4ab0-be47-623f932180a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25"
Jul 17 19:10:55.648835 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:55.648779    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25"
Jul 17 19:10:55.784082 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:55.784058    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25"]
Jul 17 19:10:55.785572 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:10:55.785547    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ffbceba_acda_4ab0_be47_623f932180a7.slice/crio-affd5fbb89d57a1eff1efadc3bc353b2e478a969270d0a33f0f72d65b58cf5e2 WatchSource:0}: Error finding container affd5fbb89d57a1eff1efadc3bc353b2e478a969270d0a33f0f72d65b58cf5e2: Status 404 returned error can't find the container with id affd5fbb89d57a1eff1efadc3bc353b2e478a969270d0a33f0f72d65b58cf5e2
Jul 17 19:10:55.953236 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:55.953212    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25" event={"ID":"8ffbceba-acda-4ab0-be47-623f932180a7","Type":"ContainerStarted","Data":"df966925d3992adf29c70252c9e70a99c918cc7b912c68f8f0e89586afbec283"}
Jul 17 19:10:55.953375 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:55.953255    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25" event={"ID":"8ffbceba-acda-4ab0-be47-623f932180a7","Type":"ContainerStarted","Data":"affd5fbb89d57a1eff1efadc3bc353b2e478a969270d0a33f0f72d65b58cf5e2"}
Jul 17 19:10:55.972773 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:55.972729    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25" podStartSLOduration=0.972713639 podStartE2EDuration="972.713639ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:10:55.971269435 +0000 UTC m=+2266.412778119" watchObservedRunningTime="2026-07-17 19:10:55.972713639 +0000 UTC m=+2266.414222328"
Jul 17 19:10:56.960316 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:56.960279    2559 generic.go:356] "Generic (PLEG): container finished" podID="8ffbceba-acda-4ab0-be47-623f932180a7" containerID="df966925d3992adf29c70252c9e70a99c918cc7b912c68f8f0e89586afbec283" exitCode=6
Jul 17 19:10:56.960726 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:56.960367    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25" event={"ID":"8ffbceba-acda-4ab0-be47-623f932180a7","Type":"ContainerDied","Data":"df966925d3992adf29c70252c9e70a99c918cc7b912c68f8f0e89586afbec283"}
Jul 17 19:10:56.960726 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:56.960548    2559 scope.go:117] "RemoveContainer" containerID="df966925d3992adf29c70252c9e70a99c918cc7b912c68f8f0e89586afbec283"
Jul 17 19:10:57.968399 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:57.968298    2559 generic.go:356] "Generic (PLEG): container finished" podID="8ffbceba-acda-4ab0-be47-623f932180a7" containerID="ce7a67dfa0ecfea7e1fa86335878582e081a759f57d7d7ba2c45185cf6ebd27f" exitCode=6
Jul 17 19:10:57.968399 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:57.968343    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25" event={"ID":"8ffbceba-acda-4ab0-be47-623f932180a7","Type":"ContainerDied","Data":"ce7a67dfa0ecfea7e1fa86335878582e081a759f57d7d7ba2c45185cf6ebd27f"}
Jul 17 19:10:57.968823 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:57.968434    2559 scope.go:117] "RemoveContainer" containerID="df966925d3992adf29c70252c9e70a99c918cc7b912c68f8f0e89586afbec283"
Jul 17 19:10:57.968823 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:57.968585    2559 scope.go:117] "RemoveContainer" containerID="ce7a67dfa0ecfea7e1fa86335878582e081a759f57d7d7ba2c45185cf6ebd27f"
Jul 17 19:10:57.968823 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:10:57.968782    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25_odh-ai-gateway-infra(8ffbceba-acda-4ab0-be47-623f932180a7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25" podUID="8ffbceba-acda-4ab0-be47-623f932180a7"
Jul 17 19:10:58.976237 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:10:58.976206    2559 scope.go:117] "RemoveContainer" containerID="ce7a67dfa0ecfea7e1fa86335878582e081a759f57d7d7ba2c45185cf6ebd27f"
Jul 17 19:10:58.976727 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:10:58.976457    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25_odh-ai-gateway-infra(8ffbceba-acda-4ab0-be47-623f932180a7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25" podUID="8ffbceba-acda-4ab0-be47-623f932180a7"
Jul 17 19:11:00.122271 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:00.122236    2559 scope.go:117] "RemoveContainer" containerID="99e8d7ccca329c58d521cef8cf78f5171bdf86f121b74ea476f73ab9efee967e"
Jul 17 19:11:00.989621 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:00.989584    2559 generic.go:356] "Generic (PLEG): container finished" podID="277174ba-8702-48d0-a3b3-cd8d57758263" containerID="64e3e85fb8aeca6f4c84a989f0a01ac4a3829d3b82906935c865942d3bfa363d" exitCode=6
Jul 17 19:11:00.989778 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:00.989660    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd" event={"ID":"277174ba-8702-48d0-a3b3-cd8d57758263","Type":"ContainerDied","Data":"64e3e85fb8aeca6f4c84a989f0a01ac4a3829d3b82906935c865942d3bfa363d"}
Jul 17 19:11:00.989778 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:00.989727    2559 scope.go:117] "RemoveContainer" containerID="99e8d7ccca329c58d521cef8cf78f5171bdf86f121b74ea476f73ab9efee967e"
Jul 17 19:11:00.990057 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:00.990031    2559 scope.go:117] "RemoveContainer" containerID="64e3e85fb8aeca6f4c84a989f0a01ac4a3829d3b82906935c865942d3bfa363d"
Jul 17 19:11:00.990281 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:11:00.990259    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd_odh-ai-gateway-infra(277174ba-8702-48d0-a3b3-cd8d57758263)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd" podUID="277174ba-8702-48d0-a3b3-cd8d57758263"
Jul 17 19:11:02.023696 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:02.023662    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd"]
Jul 17 19:11:02.175849 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:02.175826    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd"
Jul 17 19:11:02.322282 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:02.322254    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/277174ba-8702-48d0-a3b3-cd8d57758263-maas-api-service-ca\") pod \"277174ba-8702-48d0-a3b3-cd8d57758263\" (UID: \"277174ba-8702-48d0-a3b3-cd8d57758263\") "
Jul 17 19:11:02.322598 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:02.322577    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277174ba-8702-48d0-a3b3-cd8d57758263-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "277174ba-8702-48d0-a3b3-cd8d57758263" (UID: "277174ba-8702-48d0-a3b3-cd8d57758263"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:11:02.423860 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:02.423824    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/277174ba-8702-48d0-a3b3-cd8d57758263-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:11:03.003125 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:03.003092    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd" event={"ID":"277174ba-8702-48d0-a3b3-cd8d57758263","Type":"ContainerDied","Data":"f014c924854db95a1787082a011368a9e8217fe18d49b38d3878137e39175cbb"}
Jul 17 19:11:03.003125 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:03.003112    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd"
Jul 17 19:11:03.003371 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:03.003174    2559 scope.go:117] "RemoveContainer" containerID="64e3e85fb8aeca6f4c84a989f0a01ac4a3829d3b82906935c865942d3bfa363d"
Jul 17 19:11:03.026283 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:03.026258    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd"]
Jul 17 19:11:03.031071 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:03.031048    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-nq2rd"]
Jul 17 19:11:04.125733 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:04.125698    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="277174ba-8702-48d0-a3b3-cd8d57758263" path="/var/lib/kubelet/pods/277174ba-8702-48d0-a3b3-cd8d57758263/volumes"
Jul 17 19:11:14.119541 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:14.119510    2559 scope.go:117] "RemoveContainer" containerID="ce7a67dfa0ecfea7e1fa86335878582e081a759f57d7d7ba2c45185cf6ebd27f"
Jul 17 19:11:15.070115 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:15.070076    2559 generic.go:356] "Generic (PLEG): container finished" podID="8ffbceba-acda-4ab0-be47-623f932180a7" containerID="21f063e86945edb2b0dc5882361188ba30abc1faac07c09c3fde760ce981a14d" exitCode=6
Jul 17 19:11:15.070289 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:15.070116    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25" event={"ID":"8ffbceba-acda-4ab0-be47-623f932180a7","Type":"ContainerDied","Data":"21f063e86945edb2b0dc5882361188ba30abc1faac07c09c3fde760ce981a14d"}
Jul 17 19:11:15.070289 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:15.070179    2559 scope.go:117] "RemoveContainer" containerID="ce7a67dfa0ecfea7e1fa86335878582e081a759f57d7d7ba2c45185cf6ebd27f"
Jul 17 19:11:15.070523 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:15.070506    2559 scope.go:117] "RemoveContainer" containerID="21f063e86945edb2b0dc5882361188ba30abc1faac07c09c3fde760ce981a14d"
Jul 17 19:11:15.070746 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:11:15.070723    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25_odh-ai-gateway-infra(8ffbceba-acda-4ab0-be47-623f932180a7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25" podUID="8ffbceba-acda-4ab0-be47-623f932180a7"
Jul 17 19:11:16.096222 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:16.096189    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25"]
Jul 17 19:11:16.234298 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:16.234266    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25"
Jul 17 19:11:16.349456 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:16.349370    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ffbceba-acda-4ab0-be47-623f932180a7-maas-api-service-ca\") pod \"8ffbceba-acda-4ab0-be47-623f932180a7\" (UID: \"8ffbceba-acda-4ab0-be47-623f932180a7\") "
Jul 17 19:11:16.349712 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:16.349688    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ffbceba-acda-4ab0-be47-623f932180a7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ffbceba-acda-4ab0-be47-623f932180a7" (UID: "8ffbceba-acda-4ab0-be47-623f932180a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:11:16.450725 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:16.450684    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ffbceba-acda-4ab0-be47-623f932180a7-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:11:17.084622 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:17.084599    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25"
Jul 17 19:11:17.084798 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:17.084597    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25" event={"ID":"8ffbceba-acda-4ab0-be47-623f932180a7","Type":"ContainerDied","Data":"affd5fbb89d57a1eff1efadc3bc353b2e478a969270d0a33f0f72d65b58cf5e2"}
Jul 17 19:11:17.084798 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:17.084763    2559 scope.go:117] "RemoveContainer" containerID="21f063e86945edb2b0dc5882361188ba30abc1faac07c09c3fde760ce981a14d"
Jul 17 19:11:17.116247 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:17.116218    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25"]
Jul 17 19:11:17.123665 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:17.123642    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnn25"]
Jul 17 19:11:18.125361 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:18.125327    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ffbceba-acda-4ab0-be47-623f932180a7" path="/var/lib/kubelet/pods/8ffbceba-acda-4ab0-be47-623f932180a7/volumes"
Jul 17 19:11:22.394071 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.394031    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7"]
Jul 17 19:11:22.394584 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.394485    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="277174ba-8702-48d0-a3b3-cd8d57758263" containerName="revoke-keys"
Jul 17 19:11:22.394584 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.394498    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="277174ba-8702-48d0-a3b3-cd8d57758263" containerName="revoke-keys"
Jul 17 19:11:22.394584 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.394506    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="277174ba-8702-48d0-a3b3-cd8d57758263" containerName="revoke-keys"
Jul 17 19:11:22.394584 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.394511    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="277174ba-8702-48d0-a3b3-cd8d57758263" containerName="revoke-keys"
Jul 17 19:11:22.394584 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.394522    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ffbceba-acda-4ab0-be47-623f932180a7" containerName="revoke-keys"
Jul 17 19:11:22.394584 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.394527    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ffbceba-acda-4ab0-be47-623f932180a7" containerName="revoke-keys"
Jul 17 19:11:22.394584 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.394539    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ffbceba-acda-4ab0-be47-623f932180a7" containerName="revoke-keys"
Jul 17 19:11:22.394584 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.394545    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ffbceba-acda-4ab0-be47-623f932180a7" containerName="revoke-keys"
Jul 17 19:11:22.394584 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.394554    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ffbceba-acda-4ab0-be47-623f932180a7" containerName="revoke-keys"
Jul 17 19:11:22.394584 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.394562    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ffbceba-acda-4ab0-be47-623f932180a7" containerName="revoke-keys"
Jul 17 19:11:22.395102 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.394622    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ffbceba-acda-4ab0-be47-623f932180a7" containerName="revoke-keys"
Jul 17 19:11:22.395102 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.394632    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="277174ba-8702-48d0-a3b3-cd8d57758263" containerName="revoke-keys"
Jul 17 19:11:22.395102 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.394642    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="277174ba-8702-48d0-a3b3-cd8d57758263" containerName="revoke-keys"
Jul 17 19:11:22.395102 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.394653    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="277174ba-8702-48d0-a3b3-cd8d57758263" containerName="revoke-keys"
Jul 17 19:11:22.395102 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.394660    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ffbceba-acda-4ab0-be47-623f932180a7" containerName="revoke-keys"
Jul 17 19:11:22.398913 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.398892    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7"
Jul 17 19:11:22.401869 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.401721    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:11:22.405812 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.405789    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7"]
Jul 17 19:11:22.408728 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.408702    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0adb7a-db87-4e7f-a3d6-7606bf6f784e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7\" (UID: \"df0adb7a-db87-4e7f-a3d6-7606bf6f784e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7"
Jul 17 19:11:22.509880 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.509843    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0adb7a-db87-4e7f-a3d6-7606bf6f784e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7\" (UID: \"df0adb7a-db87-4e7f-a3d6-7606bf6f784e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7"
Jul 17 19:11:22.510573 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.510555    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0adb7a-db87-4e7f-a3d6-7606bf6f784e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7\" (UID: \"df0adb7a-db87-4e7f-a3d6-7606bf6f784e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7"
Jul 17 19:11:22.711397 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.711291    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7"
Jul 17 19:11:22.849006 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:22.848982    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7"]
Jul 17 19:11:22.851251 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:11:22.851224    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0adb7a_db87_4e7f_a3d6_7606bf6f784e.slice/crio-a24e16a74be884f52f72ad9fa7aefe82157472b998c9f9b0ef66fa842ba123cf WatchSource:0}: Error finding container a24e16a74be884f52f72ad9fa7aefe82157472b998c9f9b0ef66fa842ba123cf: Status 404 returned error can't find the container with id a24e16a74be884f52f72ad9fa7aefe82157472b998c9f9b0ef66fa842ba123cf
Jul 17 19:11:23.120552 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:23.120517    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7" event={"ID":"df0adb7a-db87-4e7f-a3d6-7606bf6f784e","Type":"ContainerStarted","Data":"a24e16a74be884f52f72ad9fa7aefe82157472b998c9f9b0ef66fa842ba123cf"}
Jul 17 19:11:24.127878 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:24.127846    2559 generic.go:356] "Generic (PLEG): container finished" podID="df0adb7a-db87-4e7f-a3d6-7606bf6f784e" containerID="91a329cade40ea77d1339602bb4658f810de8191196d580ed3a0bae780629354" exitCode=7
Jul 17 19:11:24.128292 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:24.127914    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7" event={"ID":"df0adb7a-db87-4e7f-a3d6-7606bf6f784e","Type":"ContainerDied","Data":"91a329cade40ea77d1339602bb4658f810de8191196d580ed3a0bae780629354"}
Jul 17 19:11:24.128292 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:24.128055    2559 scope.go:117] "RemoveContainer" containerID="91a329cade40ea77d1339602bb4658f810de8191196d580ed3a0bae780629354"
Jul 17 19:11:25.134907 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:25.134871    2559 generic.go:356] "Generic (PLEG): container finished" podID="df0adb7a-db87-4e7f-a3d6-7606bf6f784e" containerID="22f47e909c42a49c0ad560aaca0a1350f81f6579f0f18237c9d0acd5dd5d9a01" exitCode=7
Jul 17 19:11:25.135437 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:25.134933    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7" event={"ID":"df0adb7a-db87-4e7f-a3d6-7606bf6f784e","Type":"ContainerDied","Data":"22f47e909c42a49c0ad560aaca0a1350f81f6579f0f18237c9d0acd5dd5d9a01"}
Jul 17 19:11:25.135437 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:25.134985    2559 scope.go:117] "RemoveContainer" containerID="91a329cade40ea77d1339602bb4658f810de8191196d580ed3a0bae780629354"
Jul 17 19:11:25.135437 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:25.135155    2559 scope.go:117] "RemoveContainer" containerID="22f47e909c42a49c0ad560aaca0a1350f81f6579f0f18237c9d0acd5dd5d9a01"
Jul 17 19:11:25.135437 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:11:25.135412    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7_odh-ai-gateway-infra(df0adb7a-db87-4e7f-a3d6-7606bf6f784e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7" podUID="df0adb7a-db87-4e7f-a3d6-7606bf6f784e"
Jul 17 19:11:26.142106 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:26.142077    2559 scope.go:117] "RemoveContainer" containerID="22f47e909c42a49c0ad560aaca0a1350f81f6579f0f18237c9d0acd5dd5d9a01"
Jul 17 19:11:26.142581 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:11:26.142361    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7_odh-ai-gateway-infra(df0adb7a-db87-4e7f-a3d6-7606bf6f784e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7" podUID="df0adb7a-db87-4e7f-a3d6-7606bf6f784e"
Jul 17 19:11:33.548250 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:33.548216    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6"]
Jul 17 19:11:33.548798 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:33.548777    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="277174ba-8702-48d0-a3b3-cd8d57758263" containerName="revoke-keys"
Jul 17 19:11:33.548883 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:33.548801    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="277174ba-8702-48d0-a3b3-cd8d57758263" containerName="revoke-keys"
Jul 17 19:11:33.548938 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:33.548891    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ffbceba-acda-4ab0-be47-623f932180a7" containerName="revoke-keys"
Jul 17 19:11:33.553068 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:33.553046    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6"
Jul 17 19:11:33.560538 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:33.560505    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6"]
Jul 17 19:11:33.607912 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:33.607863    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ee4bf6d-b974-44ea-9797-1d44243e1756-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6\" (UID: \"2ee4bf6d-b974-44ea-9797-1d44243e1756\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6"
Jul 17 19:11:33.709128 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:33.709093    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ee4bf6d-b974-44ea-9797-1d44243e1756-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6\" (UID: \"2ee4bf6d-b974-44ea-9797-1d44243e1756\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6"
Jul 17 19:11:33.709693 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:33.709676    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ee4bf6d-b974-44ea-9797-1d44243e1756-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6\" (UID: \"2ee4bf6d-b974-44ea-9797-1d44243e1756\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6"
Jul 17 19:11:33.864644 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:33.864626    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6"
Jul 17 19:11:34.000428 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:34.000399    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6"]
Jul 17 19:11:34.002210 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:11:34.002183    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee4bf6d_b974_44ea_9797_1d44243e1756.slice/crio-8a0c2f2656b5d5c4dd56108be1f4d86585bffaded46991613e212139e5a976c1 WatchSource:0}: Error finding container 8a0c2f2656b5d5c4dd56108be1f4d86585bffaded46991613e212139e5a976c1: Status 404 returned error can't find the container with id 8a0c2f2656b5d5c4dd56108be1f4d86585bffaded46991613e212139e5a976c1
Jul 17 19:11:34.185757 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:34.185724    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6" event={"ID":"2ee4bf6d-b974-44ea-9797-1d44243e1756","Type":"ContainerStarted","Data":"8a0c2f2656b5d5c4dd56108be1f4d86585bffaded46991613e212139e5a976c1"}
Jul 17 19:11:35.192892 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:35.192854    2559 generic.go:356] "Generic (PLEG): container finished" podID="2ee4bf6d-b974-44ea-9797-1d44243e1756" containerID="66ecce6bff57551155e133dff06b56528931226fd782f3c50c95e3b265d110e1" exitCode=6
Jul 17 19:11:35.193401 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:35.192908    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6" event={"ID":"2ee4bf6d-b974-44ea-9797-1d44243e1756","Type":"ContainerDied","Data":"66ecce6bff57551155e133dff06b56528931226fd782f3c50c95e3b265d110e1"}
Jul 17 19:11:35.193401 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:35.193134    2559 scope.go:117] "RemoveContainer" containerID="66ecce6bff57551155e133dff06b56528931226fd782f3c50c95e3b265d110e1"
Jul 17 19:11:36.199687 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:36.199595    2559 generic.go:356] "Generic (PLEG): container finished" podID="2ee4bf6d-b974-44ea-9797-1d44243e1756" containerID="3350cf1c10aaa00b7cb8a0899da8983eec34b8155f7fc4ba55e7befcd7e63fc8" exitCode=6
Jul 17 19:11:36.199687 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:36.199665    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6" event={"ID":"2ee4bf6d-b974-44ea-9797-1d44243e1756","Type":"ContainerDied","Data":"3350cf1c10aaa00b7cb8a0899da8983eec34b8155f7fc4ba55e7befcd7e63fc8"}
Jul 17 19:11:36.200166 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:36.199714    2559 scope.go:117] "RemoveContainer" containerID="66ecce6bff57551155e133dff06b56528931226fd782f3c50c95e3b265d110e1"
Jul 17 19:11:36.200166 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:36.199896    2559 scope.go:117] "RemoveContainer" containerID="3350cf1c10aaa00b7cb8a0899da8983eec34b8155f7fc4ba55e7befcd7e63fc8"
Jul 17 19:11:36.200166 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:11:36.200098    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6_odh-ai-gateway-infra(2ee4bf6d-b974-44ea-9797-1d44243e1756)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6" podUID="2ee4bf6d-b974-44ea-9797-1d44243e1756"
Jul 17 19:11:37.206940 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:37.206910    2559 scope.go:117] "RemoveContainer" containerID="3350cf1c10aaa00b7cb8a0899da8983eec34b8155f7fc4ba55e7befcd7e63fc8"
Jul 17 19:11:37.207337 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:11:37.207095    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6_odh-ai-gateway-infra(2ee4bf6d-b974-44ea-9797-1d44243e1756)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6" podUID="2ee4bf6d-b974-44ea-9797-1d44243e1756"
Jul 17 19:11:38.119561 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:38.119525    2559 scope.go:117] "RemoveContainer" containerID="22f47e909c42a49c0ad560aaca0a1350f81f6579f0f18237c9d0acd5dd5d9a01"
Jul 17 19:11:39.218322 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:39.218276    2559 generic.go:356] "Generic (PLEG): container finished" podID="df0adb7a-db87-4e7f-a3d6-7606bf6f784e" containerID="485dcffc0926da710c4f9793ff7fbd73b92d31db97389359c05f9e83016333f3" exitCode=7
Jul 17 19:11:39.218706 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:39.218346    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7" event={"ID":"df0adb7a-db87-4e7f-a3d6-7606bf6f784e","Type":"ContainerDied","Data":"485dcffc0926da710c4f9793ff7fbd73b92d31db97389359c05f9e83016333f3"}
Jul 17 19:11:39.218706 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:39.218413    2559 scope.go:117] "RemoveContainer" containerID="22f47e909c42a49c0ad560aaca0a1350f81f6579f0f18237c9d0acd5dd5d9a01"
Jul 17 19:11:39.218706 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:39.218647    2559 scope.go:117] "RemoveContainer" containerID="485dcffc0926da710c4f9793ff7fbd73b92d31db97389359c05f9e83016333f3"
Jul 17 19:11:39.218889 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:11:39.218862    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7_odh-ai-gateway-infra(df0adb7a-db87-4e7f-a3d6-7606bf6f784e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7" podUID="df0adb7a-db87-4e7f-a3d6-7606bf6f784e"
Jul 17 19:11:40.245816 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:40.245777    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7"]
Jul 17 19:11:40.381918 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:40.381892    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7"
Jul 17 19:11:40.468215 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:40.468184    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0adb7a-db87-4e7f-a3d6-7606bf6f784e-maas-api-service-ca\") pod \"df0adb7a-db87-4e7f-a3d6-7606bf6f784e\" (UID: \"df0adb7a-db87-4e7f-a3d6-7606bf6f784e\") "
Jul 17 19:11:40.468537 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:40.468505    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0adb7a-db87-4e7f-a3d6-7606bf6f784e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "df0adb7a-db87-4e7f-a3d6-7606bf6f784e" (UID: "df0adb7a-db87-4e7f-a3d6-7606bf6f784e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:11:40.569785 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:40.569758    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0adb7a-db87-4e7f-a3d6-7606bf6f784e-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:11:41.232591 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:41.232536    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7" event={"ID":"df0adb7a-db87-4e7f-a3d6-7606bf6f784e","Type":"ContainerDied","Data":"a24e16a74be884f52f72ad9fa7aefe82157472b998c9f9b0ef66fa842ba123cf"}
Jul 17 19:11:41.232591 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:41.232586    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7"
Jul 17 19:11:41.232813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:41.232626    2559 scope.go:117] "RemoveContainer" containerID="485dcffc0926da710c4f9793ff7fbd73b92d31db97389359c05f9e83016333f3"
Jul 17 19:11:41.256909 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:41.256884    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7"]
Jul 17 19:11:41.263859 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:41.263837    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-hlkj7"]
Jul 17 19:11:42.125169 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:42.125136    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df0adb7a-db87-4e7f-a3d6-7606bf6f784e" path="/var/lib/kubelet/pods/df0adb7a-db87-4e7f-a3d6-7606bf6f784e/volumes"
Jul 17 19:11:51.119569 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:51.119532    2559 scope.go:117] "RemoveContainer" containerID="3350cf1c10aaa00b7cb8a0899da8983eec34b8155f7fc4ba55e7befcd7e63fc8"
Jul 17 19:11:52.295648 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:52.295615    2559 generic.go:356] "Generic (PLEG): container finished" podID="2ee4bf6d-b974-44ea-9797-1d44243e1756" containerID="c07d9e6b50c81d5a8376139ca999b6b733900168a04f4a2b98c558f08216ad76" exitCode=6
Jul 17 19:11:52.296064 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:52.295691    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6" event={"ID":"2ee4bf6d-b974-44ea-9797-1d44243e1756","Type":"ContainerDied","Data":"c07d9e6b50c81d5a8376139ca999b6b733900168a04f4a2b98c558f08216ad76"}
Jul 17 19:11:52.296064 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:52.295762    2559 scope.go:117] "RemoveContainer" containerID="3350cf1c10aaa00b7cb8a0899da8983eec34b8155f7fc4ba55e7befcd7e63fc8"
Jul 17 19:11:52.296185 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:52.296066    2559 scope.go:117] "RemoveContainer" containerID="c07d9e6b50c81d5a8376139ca999b6b733900168a04f4a2b98c558f08216ad76"
Jul 17 19:11:52.296278 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:11:52.296257    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6_odh-ai-gateway-infra(2ee4bf6d-b974-44ea-9797-1d44243e1756)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6" podUID="2ee4bf6d-b974-44ea-9797-1d44243e1756"
Jul 17 19:11:53.322211 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:53.322179    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6"]
Jul 17 19:11:53.461075 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:53.461054    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6"
Jul 17 19:11:53.580068 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:53.579992    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ee4bf6d-b974-44ea-9797-1d44243e1756-maas-api-service-ca\") pod \"2ee4bf6d-b974-44ea-9797-1d44243e1756\" (UID: \"2ee4bf6d-b974-44ea-9797-1d44243e1756\") "
Jul 17 19:11:53.580333 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:53.580297    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ee4bf6d-b974-44ea-9797-1d44243e1756-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ee4bf6d-b974-44ea-9797-1d44243e1756" (UID: "2ee4bf6d-b974-44ea-9797-1d44243e1756"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:11:53.681459 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:53.681429    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ee4bf6d-b974-44ea-9797-1d44243e1756-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:11:54.310643 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:54.310616    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6"
Jul 17 19:11:54.310806 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:54.310640    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6" event={"ID":"2ee4bf6d-b974-44ea-9797-1d44243e1756","Type":"ContainerDied","Data":"8a0c2f2656b5d5c4dd56108be1f4d86585bffaded46991613e212139e5a976c1"}
Jul 17 19:11:54.310806 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:54.310695    2559 scope.go:117] "RemoveContainer" containerID="c07d9e6b50c81d5a8376139ca999b6b733900168a04f4a2b98c558f08216ad76"
Jul 17 19:11:54.327780 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:54.327757    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6"]
Jul 17 19:11:54.336976 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:54.336956    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-m6qc6"]
Jul 17 19:11:55.375936 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.375908    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml"]
Jul 17 19:11:55.376298 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.376269    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0adb7a-db87-4e7f-a3d6-7606bf6f784e" containerName="revoke-keys"
Jul 17 19:11:55.376298 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.376281    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0adb7a-db87-4e7f-a3d6-7606bf6f784e" containerName="revoke-keys"
Jul 17 19:11:55.376298 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.376294    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0adb7a-db87-4e7f-a3d6-7606bf6f784e" containerName="revoke-keys"
Jul 17 19:11:55.376465 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.376301    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0adb7a-db87-4e7f-a3d6-7606bf6f784e" containerName="revoke-keys"
Jul 17 19:11:55.376465 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.376338    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ee4bf6d-b974-44ea-9797-1d44243e1756" containerName="revoke-keys"
Jul 17 19:11:55.376465 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.376344    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ee4bf6d-b974-44ea-9797-1d44243e1756" containerName="revoke-keys"
Jul 17 19:11:55.376465 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.376352    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ee4bf6d-b974-44ea-9797-1d44243e1756" containerName="revoke-keys"
Jul 17 19:11:55.376465 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.376357    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ee4bf6d-b974-44ea-9797-1d44243e1756" containerName="revoke-keys"
Jul 17 19:11:55.376465 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.376369    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ee4bf6d-b974-44ea-9797-1d44243e1756" containerName="revoke-keys"
Jul 17 19:11:55.376465 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.376374    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ee4bf6d-b974-44ea-9797-1d44243e1756" containerName="revoke-keys"
Jul 17 19:11:55.376465 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.376429    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="df0adb7a-db87-4e7f-a3d6-7606bf6f784e" containerName="revoke-keys"
Jul 17 19:11:55.376465 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.376436    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="df0adb7a-db87-4e7f-a3d6-7606bf6f784e" containerName="revoke-keys"
Jul 17 19:11:55.376465 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.376442    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ee4bf6d-b974-44ea-9797-1d44243e1756" containerName="revoke-keys"
Jul 17 19:11:55.376465 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.376450    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ee4bf6d-b974-44ea-9797-1d44243e1756" containerName="revoke-keys"
Jul 17 19:11:55.376465 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.376456    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ee4bf6d-b974-44ea-9797-1d44243e1756" containerName="revoke-keys"
Jul 17 19:11:55.380831 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.380813    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml"
Jul 17 19:11:55.384135 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.384115    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:11:55.387450 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.387428    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml"]
Jul 17 19:11:55.494828 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.494792    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13871bd-2d26-4fd9-bb96-95b071e12e56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml\" (UID: \"c13871bd-2d26-4fd9-bb96-95b071e12e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml"
Jul 17 19:11:55.595662 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.595636    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13871bd-2d26-4fd9-bb96-95b071e12e56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml\" (UID: \"c13871bd-2d26-4fd9-bb96-95b071e12e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml"
Jul 17 19:11:55.596204 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.596184    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13871bd-2d26-4fd9-bb96-95b071e12e56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml\" (UID: \"c13871bd-2d26-4fd9-bb96-95b071e12e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml"
Jul 17 19:11:55.693217 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.693165    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml"
Jul 17 19:11:55.827733 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:55.827708    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml"]
Jul 17 19:11:55.829217 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:11:55.829192    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13871bd_2d26_4fd9_bb96_95b071e12e56.slice/crio-5596d32a02acb84dbeb7538f78d4ec9a5d6bc80d212add27ae6aaf20a54cfcb7 WatchSource:0}: Error finding container 5596d32a02acb84dbeb7538f78d4ec9a5d6bc80d212add27ae6aaf20a54cfcb7: Status 404 returned error can't find the container with id 5596d32a02acb84dbeb7538f78d4ec9a5d6bc80d212add27ae6aaf20a54cfcb7
Jul 17 19:11:56.124963 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:56.124930    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ee4bf6d-b974-44ea-9797-1d44243e1756" path="/var/lib/kubelet/pods/2ee4bf6d-b974-44ea-9797-1d44243e1756/volumes"
Jul 17 19:11:56.324229 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:56.324197    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml" event={"ID":"c13871bd-2d26-4fd9-bb96-95b071e12e56","Type":"ContainerStarted","Data":"772a1427a050ae26c3d072659b7d082a95a31c511247de0c2adcd199820d7bce"}
Jul 17 19:11:56.324377 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:56.324250    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml" event={"ID":"c13871bd-2d26-4fd9-bb96-95b071e12e56","Type":"ContainerStarted","Data":"5596d32a02acb84dbeb7538f78d4ec9a5d6bc80d212add27ae6aaf20a54cfcb7"}
Jul 17 19:11:56.342692 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:56.342647    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml" podStartSLOduration=1.342634181 podStartE2EDuration="1.342634181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:11:56.339233088 +0000 UTC m=+2326.780741778" watchObservedRunningTime="2026-07-17 19:11:56.342634181 +0000 UTC m=+2326.784142870"
Jul 17 19:11:57.330635 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:57.330599    2559 generic.go:356] "Generic (PLEG): container finished" podID="c13871bd-2d26-4fd9-bb96-95b071e12e56" containerID="772a1427a050ae26c3d072659b7d082a95a31c511247de0c2adcd199820d7bce" exitCode=6
Jul 17 19:11:57.331077 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:57.330690    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml" event={"ID":"c13871bd-2d26-4fd9-bb96-95b071e12e56","Type":"ContainerDied","Data":"772a1427a050ae26c3d072659b7d082a95a31c511247de0c2adcd199820d7bce"}
Jul 17 19:11:57.331077 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:57.330850    2559 scope.go:117] "RemoveContainer" containerID="772a1427a050ae26c3d072659b7d082a95a31c511247de0c2adcd199820d7bce"
Jul 17 19:11:58.338039 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:58.338009    2559 generic.go:356] "Generic (PLEG): container finished" podID="c13871bd-2d26-4fd9-bb96-95b071e12e56" containerID="1c01d51e42eb4f9405a86e053483519e5f37c5af13ff1e78efd727f39ceae967" exitCode=6
Jul 17 19:11:58.338493 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:58.338092    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml" event={"ID":"c13871bd-2d26-4fd9-bb96-95b071e12e56","Type":"ContainerDied","Data":"1c01d51e42eb4f9405a86e053483519e5f37c5af13ff1e78efd727f39ceae967"}
Jul 17 19:11:58.338493 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:58.338154    2559 scope.go:117] "RemoveContainer" containerID="772a1427a050ae26c3d072659b7d082a95a31c511247de0c2adcd199820d7bce"
Jul 17 19:11:58.338493 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:58.338349    2559 scope.go:117] "RemoveContainer" containerID="1c01d51e42eb4f9405a86e053483519e5f37c5af13ff1e78efd727f39ceae967"
Jul 17 19:11:58.338654 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:11:58.338575    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml_odh-ai-gateway-infra(c13871bd-2d26-4fd9-bb96-95b071e12e56)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml" podUID="c13871bd-2d26-4fd9-bb96-95b071e12e56"
Jul 17 19:11:59.345241 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:11:59.345212    2559 scope.go:117] "RemoveContainer" containerID="1c01d51e42eb4f9405a86e053483519e5f37c5af13ff1e78efd727f39ceae967"
Jul 17 19:11:59.345655 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:11:59.345407    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml_odh-ai-gateway-infra(c13871bd-2d26-4fd9-bb96-95b071e12e56)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml" podUID="c13871bd-2d26-4fd9-bb96-95b071e12e56"
Jul 17 19:12:12.435162 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:12.435114    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6"]
Jul 17 19:12:12.435577 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:12.435560    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0adb7a-db87-4e7f-a3d6-7606bf6f784e" containerName="revoke-keys"
Jul 17 19:12:12.435577 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:12.435577    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0adb7a-db87-4e7f-a3d6-7606bf6f784e" containerName="revoke-keys"
Jul 17 19:12:12.435652 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:12.435640    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="df0adb7a-db87-4e7f-a3d6-7606bf6f784e" containerName="revoke-keys"
Jul 17 19:12:12.440345 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:12.440326    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6"
Jul 17 19:12:12.448498 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:12.448105    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6"]
Jul 17 19:12:12.537706 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:12.537662    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6\" (UID: \"8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6"
Jul 17 19:12:12.638725 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:12.638686    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6\" (UID: \"8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6"
Jul 17 19:12:12.639275 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:12.639256    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6\" (UID: \"8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6"
Jul 17 19:12:12.752499 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:12.752420    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6"
Jul 17 19:12:12.888493 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:12.888461    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6"]
Jul 17 19:12:12.890010 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:12:12.889979    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d54eb43_3eb3_46d7_b52d_ab15cf88e7d4.slice/crio-bf0a11e8376b11e1acc3166faff8e352e9782b90faa64342d3745afd49f4bc26 WatchSource:0}: Error finding container bf0a11e8376b11e1acc3166faff8e352e9782b90faa64342d3745afd49f4bc26: Status 404 returned error can't find the container with id bf0a11e8376b11e1acc3166faff8e352e9782b90faa64342d3745afd49f4bc26
Jul 17 19:12:13.119423 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:13.119396    2559 scope.go:117] "RemoveContainer" containerID="1c01d51e42eb4f9405a86e053483519e5f37c5af13ff1e78efd727f39ceae967"
Jul 17 19:12:13.425486 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:13.425450    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6" event={"ID":"8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4","Type":"ContainerStarted","Data":"520137c6710b114e4c7e90779bd14c06101a485e3561c793ef08df6aa2c7b77e"}
Jul 17 19:12:13.425700 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:13.425513    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6" event={"ID":"8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4","Type":"ContainerStarted","Data":"bf0a11e8376b11e1acc3166faff8e352e9782b90faa64342d3745afd49f4bc26"}
Jul 17 19:12:13.427438 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:13.427410    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml" event={"ID":"c13871bd-2d26-4fd9-bb96-95b071e12e56","Type":"ContainerStarted","Data":"68dc7f7a5779e7b628a6250690c50fc31ac9f176e680497d84dd898d27e8e942"}
Jul 17 19:12:13.447195 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:13.447151    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6" podStartSLOduration=1.447137366 podStartE2EDuration="1.447137366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:12:13.44640679 +0000 UTC m=+2343.887915480" watchObservedRunningTime="2026-07-17 19:12:13.447137366 +0000 UTC m=+2343.888646055"
Jul 17 19:12:14.440953 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:14.440913    2559 generic.go:356] "Generic (PLEG): container finished" podID="8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4" containerID="520137c6710b114e4c7e90779bd14c06101a485e3561c793ef08df6aa2c7b77e" exitCode=7
Jul 17 19:12:14.441168 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:14.440948    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6" event={"ID":"8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4","Type":"ContainerDied","Data":"520137c6710b114e4c7e90779bd14c06101a485e3561c793ef08df6aa2c7b77e"}
Jul 17 19:12:14.441264 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:14.441247    2559 scope.go:117] "RemoveContainer" containerID="520137c6710b114e4c7e90779bd14c06101a485e3561c793ef08df6aa2c7b77e"
Jul 17 19:12:14.442949 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:14.442931    2559 generic.go:356] "Generic (PLEG): container finished" podID="c13871bd-2d26-4fd9-bb96-95b071e12e56" containerID="68dc7f7a5779e7b628a6250690c50fc31ac9f176e680497d84dd898d27e8e942" exitCode=6
Jul 17 19:12:14.443026 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:14.443002    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml" event={"ID":"c13871bd-2d26-4fd9-bb96-95b071e12e56","Type":"ContainerDied","Data":"68dc7f7a5779e7b628a6250690c50fc31ac9f176e680497d84dd898d27e8e942"}
Jul 17 19:12:14.443063 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:14.443050    2559 scope.go:117] "RemoveContainer" containerID="1c01d51e42eb4f9405a86e053483519e5f37c5af13ff1e78efd727f39ceae967"
Jul 17 19:12:14.443262 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:14.443245    2559 scope.go:117] "RemoveContainer" containerID="68dc7f7a5779e7b628a6250690c50fc31ac9f176e680497d84dd898d27e8e942"
Jul 17 19:12:14.443476 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:12:14.443455    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml_odh-ai-gateway-infra(c13871bd-2d26-4fd9-bb96-95b071e12e56)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml" podUID="c13871bd-2d26-4fd9-bb96-95b071e12e56"
Jul 17 19:12:14.473753 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:14.473728    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml"]
Jul 17 19:12:15.450623 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:15.450589    2559 generic.go:356] "Generic (PLEG): container finished" podID="8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4" containerID="ee8fa7a4e2d8e97934a7c232000cea4707f852811203da05fc6efd8213e642f1" exitCode=7
Jul 17 19:12:15.450823 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:15.450676    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6" event={"ID":"8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4","Type":"ContainerDied","Data":"ee8fa7a4e2d8e97934a7c232000cea4707f852811203da05fc6efd8213e642f1"}
Jul 17 19:12:15.450823 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:15.450739    2559 scope.go:117] "RemoveContainer" containerID="520137c6710b114e4c7e90779bd14c06101a485e3561c793ef08df6aa2c7b77e"
Jul 17 19:12:15.451001 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:15.450952    2559 scope.go:117] "RemoveContainer" containerID="ee8fa7a4e2d8e97934a7c232000cea4707f852811203da05fc6efd8213e642f1"
Jul 17 19:12:15.451242 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:12:15.451219    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6_odh-ai-gateway-infra(8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6" podUID="8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4"
Jul 17 19:12:15.597267 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:15.597244    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml"
Jul 17 19:12:15.664728 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:15.664693    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13871bd-2d26-4fd9-bb96-95b071e12e56-maas-api-service-ca\") pod \"c13871bd-2d26-4fd9-bb96-95b071e12e56\" (UID: \"c13871bd-2d26-4fd9-bb96-95b071e12e56\") "
Jul 17 19:12:15.665125 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:15.665092    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13871bd-2d26-4fd9-bb96-95b071e12e56-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c13871bd-2d26-4fd9-bb96-95b071e12e56" (UID: "c13871bd-2d26-4fd9-bb96-95b071e12e56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:12:15.766068 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:15.765981    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13871bd-2d26-4fd9-bb96-95b071e12e56-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:12:16.461761 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:16.461681    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml"
Jul 17 19:12:16.461761 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:16.461713    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml" event={"ID":"c13871bd-2d26-4fd9-bb96-95b071e12e56","Type":"ContainerDied","Data":"5596d32a02acb84dbeb7538f78d4ec9a5d6bc80d212add27ae6aaf20a54cfcb7"}
Jul 17 19:12:16.461996 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:16.461787    2559 scope.go:117] "RemoveContainer" containerID="68dc7f7a5779e7b628a6250690c50fc31ac9f176e680497d84dd898d27e8e942"
Jul 17 19:12:16.463918 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:16.463897    2559 scope.go:117] "RemoveContainer" containerID="ee8fa7a4e2d8e97934a7c232000cea4707f852811203da05fc6efd8213e642f1"
Jul 17 19:12:16.464151 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:12:16.464130    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6_odh-ai-gateway-infra(8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6" podUID="8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4"
Jul 17 19:12:16.489605 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:16.489582    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml"]
Jul 17 19:12:16.496200 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:16.496178    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89wml"]
Jul 17 19:12:18.125779 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:18.125747    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c13871bd-2d26-4fd9-bb96-95b071e12e56" path="/var/lib/kubelet/pods/c13871bd-2d26-4fd9-bb96-95b071e12e56/volumes"
Jul 17 19:12:30.123385 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:30.123352    2559 scope.go:117] "RemoveContainer" containerID="ee8fa7a4e2d8e97934a7c232000cea4707f852811203da05fc6efd8213e642f1"
Jul 17 19:12:30.541560 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:30.541488    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6" event={"ID":"8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4","Type":"ContainerStarted","Data":"652646c00a2e066019177a44334a5f7b19ddb07aa3ac3de2835f5dec05cfceb6"}
Jul 17 19:12:31.549147 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:31.549110    2559 generic.go:356] "Generic (PLEG): container finished" podID="8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4" containerID="652646c00a2e066019177a44334a5f7b19ddb07aa3ac3de2835f5dec05cfceb6" exitCode=7
Jul 17 19:12:31.549612 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:31.549175    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6" event={"ID":"8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4","Type":"ContainerDied","Data":"652646c00a2e066019177a44334a5f7b19ddb07aa3ac3de2835f5dec05cfceb6"}
Jul 17 19:12:31.549612 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:31.549259    2559 scope.go:117] "RemoveContainer" containerID="ee8fa7a4e2d8e97934a7c232000cea4707f852811203da05fc6efd8213e642f1"
Jul 17 19:12:31.549612 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:31.549523    2559 scope.go:117] "RemoveContainer" containerID="652646c00a2e066019177a44334a5f7b19ddb07aa3ac3de2835f5dec05cfceb6"
Jul 17 19:12:31.549756 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:12:31.549736    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6_odh-ai-gateway-infra(8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6" podUID="8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4"
Jul 17 19:12:31.573671 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:31.573637    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6"]
Jul 17 19:12:32.696724 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:32.696701    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6"
Jul 17 19:12:32.815473 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:32.815452    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4-maas-api-service-ca\") pod \"8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4\" (UID: \"8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4\") "
Jul 17 19:12:32.815813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:32.815794    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4" (UID: "8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:12:32.916412 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:32.916393    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:12:33.562925 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.562900    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6"
Jul 17 19:12:33.562925 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.562903    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6" event={"ID":"8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4","Type":"ContainerDied","Data":"bf0a11e8376b11e1acc3166faff8e352e9782b90faa64342d3745afd49f4bc26"}
Jul 17 19:12:33.563162 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.562969    2559 scope.go:117] "RemoveContainer" containerID="652646c00a2e066019177a44334a5f7b19ddb07aa3ac3de2835f5dec05cfceb6"
Jul 17 19:12:33.589168 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.589140    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6"]
Jul 17 19:12:33.595425 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.595398    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc"]
Jul 17 19:12:33.595790 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.595775    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13871bd-2d26-4fd9-bb96-95b071e12e56" containerName="revoke-keys"
Jul 17 19:12:33.595855 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.595793    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13871bd-2d26-4fd9-bb96-95b071e12e56" containerName="revoke-keys"
Jul 17 19:12:33.595855 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.595806    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4" containerName="revoke-keys"
Jul 17 19:12:33.595855 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.595812    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4" containerName="revoke-keys"
Jul 17 19:12:33.595855 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.595820    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13871bd-2d26-4fd9-bb96-95b071e12e56" containerName="revoke-keys"
Jul 17 19:12:33.595855 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.595825    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13871bd-2d26-4fd9-bb96-95b071e12e56" containerName="revoke-keys"
Jul 17 19:12:33.595855 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.595832    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13871bd-2d26-4fd9-bb96-95b071e12e56" containerName="revoke-keys"
Jul 17 19:12:33.595855 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.595837    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13871bd-2d26-4fd9-bb96-95b071e12e56" containerName="revoke-keys"
Jul 17 19:12:33.595855 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.595851    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4" containerName="revoke-keys"
Jul 17 19:12:33.595855 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.595856    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4" containerName="revoke-keys"
Jul 17 19:12:33.596126 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.595912    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13871bd-2d26-4fd9-bb96-95b071e12e56" containerName="revoke-keys"
Jul 17 19:12:33.596126 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.595919    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4" containerName="revoke-keys"
Jul 17 19:12:33.596126 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.595928    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13871bd-2d26-4fd9-bb96-95b071e12e56" containerName="revoke-keys"
Jul 17 19:12:33.596126 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.595934    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13871bd-2d26-4fd9-bb96-95b071e12e56" containerName="revoke-keys"
Jul 17 19:12:33.596126 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.595939    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4" containerName="revoke-keys"
Jul 17 19:12:33.600013 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.599995    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-kzxq6"]
Jul 17 19:12:33.600094 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.600075    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc"
Jul 17 19:12:33.602485 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.602465    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:12:33.605694 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.605668    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc"]
Jul 17 19:12:33.723154 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.723120    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea755110-0a74-4fcf-9468-9787abf9f434-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc\" (UID: \"ea755110-0a74-4fcf-9468-9787abf9f434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc"
Jul 17 19:12:33.824516 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.824417    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea755110-0a74-4fcf-9468-9787abf9f434-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc\" (UID: \"ea755110-0a74-4fcf-9468-9787abf9f434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc"
Jul 17 19:12:33.825025 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.825006    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea755110-0a74-4fcf-9468-9787abf9f434-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc\" (UID: \"ea755110-0a74-4fcf-9468-9787abf9f434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc"
Jul 17 19:12:33.913820 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:33.913793    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc"
Jul 17 19:12:34.044999 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:34.044970    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc"]
Jul 17 19:12:34.046163 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:12:34.046140    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea755110_0a74_4fcf_9468_9787abf9f434.slice/crio-25b732bd0ca5a8e026ca39acc03865890e07b7ec35bfca93eded2775c42598db WatchSource:0}: Error finding container 25b732bd0ca5a8e026ca39acc03865890e07b7ec35bfca93eded2775c42598db: Status 404 returned error can't find the container with id 25b732bd0ca5a8e026ca39acc03865890e07b7ec35bfca93eded2775c42598db
Jul 17 19:12:34.125094 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:34.125063    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4" path="/var/lib/kubelet/pods/8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4/volumes"
Jul 17 19:12:34.569497 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:34.569453    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc" event={"ID":"ea755110-0a74-4fcf-9468-9787abf9f434","Type":"ContainerStarted","Data":"2dcc0f6937add98a05c167328bd7a0f651a91a6ba4d3b7432f93508c2b1deaf7"}
Jul 17 19:12:34.569696 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:34.569521    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc" event={"ID":"ea755110-0a74-4fcf-9468-9787abf9f434","Type":"ContainerStarted","Data":"25b732bd0ca5a8e026ca39acc03865890e07b7ec35bfca93eded2775c42598db"}
Jul 17 19:12:34.587944 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:34.587848    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc" podStartSLOduration=1.58783354 podStartE2EDuration="1.58783354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:12:34.586024103 +0000 UTC m=+2365.027532793" watchObservedRunningTime="2026-07-17 19:12:34.58783354 +0000 UTC m=+2365.029342228"
Jul 17 19:12:35.577005 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:35.576973    2559 generic.go:356] "Generic (PLEG): container finished" podID="ea755110-0a74-4fcf-9468-9787abf9f434" containerID="2dcc0f6937add98a05c167328bd7a0f651a91a6ba4d3b7432f93508c2b1deaf7" exitCode=6
Jul 17 19:12:35.577473 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:35.577067    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc" event={"ID":"ea755110-0a74-4fcf-9468-9787abf9f434","Type":"ContainerDied","Data":"2dcc0f6937add98a05c167328bd7a0f651a91a6ba4d3b7432f93508c2b1deaf7"}
Jul 17 19:12:35.577473 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:35.577215    2559 scope.go:117] "RemoveContainer" containerID="2dcc0f6937add98a05c167328bd7a0f651a91a6ba4d3b7432f93508c2b1deaf7"
Jul 17 19:12:36.584228 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:36.584192    2559 generic.go:356] "Generic (PLEG): container finished" podID="ea755110-0a74-4fcf-9468-9787abf9f434" containerID="552585dca74e7893f2d6e4c5af53a53923af77fcd537aa99c61d1e5aeb4aecd5" exitCode=6
Jul 17 19:12:36.584228 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:36.584228    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc" event={"ID":"ea755110-0a74-4fcf-9468-9787abf9f434","Type":"ContainerDied","Data":"552585dca74e7893f2d6e4c5af53a53923af77fcd537aa99c61d1e5aeb4aecd5"}
Jul 17 19:12:36.584781 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:36.584275    2559 scope.go:117] "RemoveContainer" containerID="2dcc0f6937add98a05c167328bd7a0f651a91a6ba4d3b7432f93508c2b1deaf7"
Jul 17 19:12:36.584781 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:36.584539    2559 scope.go:117] "RemoveContainer" containerID="552585dca74e7893f2d6e4c5af53a53923af77fcd537aa99c61d1e5aeb4aecd5"
Jul 17 19:12:36.584888 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:12:36.584785    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc_odh-ai-gateway-infra(ea755110-0a74-4fcf-9468-9787abf9f434)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc" podUID="ea755110-0a74-4fcf-9468-9787abf9f434"
Jul 17 19:12:37.591578 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:37.591546    2559 scope.go:117] "RemoveContainer" containerID="552585dca74e7893f2d6e4c5af53a53923af77fcd537aa99c61d1e5aeb4aecd5"
Jul 17 19:12:37.591965 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:12:37.591750    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc_odh-ai-gateway-infra(ea755110-0a74-4fcf-9468-9787abf9f434)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc" podUID="ea755110-0a74-4fcf-9468-9787abf9f434"
Jul 17 19:12:50.122785 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:50.122758    2559 scope.go:117] "RemoveContainer" containerID="552585dca74e7893f2d6e4c5af53a53923af77fcd537aa99c61d1e5aeb4aecd5"
Jul 17 19:12:50.664868 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:50.664835    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc" event={"ID":"ea755110-0a74-4fcf-9468-9787abf9f434","Type":"ContainerStarted","Data":"164187a53c8c3a4ae9dbe7b545df819f50fe9135c7ae7c8d81481cc9fd9fed92"}
Jul 17 19:12:51.671810 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:51.671775    2559 generic.go:356] "Generic (PLEG): container finished" podID="ea755110-0a74-4fcf-9468-9787abf9f434" containerID="164187a53c8c3a4ae9dbe7b545df819f50fe9135c7ae7c8d81481cc9fd9fed92" exitCode=6
Jul 17 19:12:51.672231 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:51.671847    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc" event={"ID":"ea755110-0a74-4fcf-9468-9787abf9f434","Type":"ContainerDied","Data":"164187a53c8c3a4ae9dbe7b545df819f50fe9135c7ae7c8d81481cc9fd9fed92"}
Jul 17 19:12:51.672231 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:51.671936    2559 scope.go:117] "RemoveContainer" containerID="552585dca74e7893f2d6e4c5af53a53923af77fcd537aa99c61d1e5aeb4aecd5"
Jul 17 19:12:51.672389 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:51.672254    2559 scope.go:117] "RemoveContainer" containerID="164187a53c8c3a4ae9dbe7b545df819f50fe9135c7ae7c8d81481cc9fd9fed92"
Jul 17 19:12:51.672547 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:12:51.672524    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc_odh-ai-gateway-infra(ea755110-0a74-4fcf-9468-9787abf9f434)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc" podUID="ea755110-0a74-4fcf-9468-9787abf9f434"
Jul 17 19:12:51.696457 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:51.696435    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc"]
Jul 17 19:12:52.813009 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:52.812985    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc"
Jul 17 19:12:52.895758 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:52.895723    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea755110-0a74-4fcf-9468-9787abf9f434-maas-api-service-ca\") pod \"ea755110-0a74-4fcf-9468-9787abf9f434\" (UID: \"ea755110-0a74-4fcf-9468-9787abf9f434\") "
Jul 17 19:12:52.896039 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:52.896020    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea755110-0a74-4fcf-9468-9787abf9f434-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ea755110-0a74-4fcf-9468-9787abf9f434" (UID: "ea755110-0a74-4fcf-9468-9787abf9f434"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:12:52.996476 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:52.996390    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea755110-0a74-4fcf-9468-9787abf9f434-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:12:53.684584 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:53.684557    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc"
Jul 17 19:12:53.684780 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:53.684606    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc" event={"ID":"ea755110-0a74-4fcf-9468-9787abf9f434","Type":"ContainerDied","Data":"25b732bd0ca5a8e026ca39acc03865890e07b7ec35bfca93eded2775c42598db"}
Jul 17 19:12:53.684780 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:53.684667    2559 scope.go:117] "RemoveContainer" containerID="164187a53c8c3a4ae9dbe7b545df819f50fe9135c7ae7c8d81481cc9fd9fed92"
Jul 17 19:12:53.713055 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:53.713031    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc"]
Jul 17 19:12:53.720152 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:53.720130    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-wjjvc"]
Jul 17 19:12:54.125895 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:54.125867    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea755110-0a74-4fcf-9468-9787abf9f434" path="/var/lib/kubelet/pods/ea755110-0a74-4fcf-9468-9787abf9f434/volumes"
Jul 17 19:12:55.421131 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:55.421098    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg"]
Jul 17 19:12:55.421534 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:55.421489    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea755110-0a74-4fcf-9468-9787abf9f434" containerName="revoke-keys"
Jul 17 19:12:55.421534 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:55.421501    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea755110-0a74-4fcf-9468-9787abf9f434" containerName="revoke-keys"
Jul 17 19:12:55.421534 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:55.421510    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea755110-0a74-4fcf-9468-9787abf9f434" containerName="revoke-keys"
Jul 17 19:12:55.421534 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:55.421516    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea755110-0a74-4fcf-9468-9787abf9f434" containerName="revoke-keys"
Jul 17 19:12:55.421746 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:55.421541    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4" containerName="revoke-keys"
Jul 17 19:12:55.421746 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:55.421547    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4" containerName="revoke-keys"
Jul 17 19:12:55.421746 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:55.421601    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea755110-0a74-4fcf-9468-9787abf9f434" containerName="revoke-keys"
Jul 17 19:12:55.421746 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:55.421609    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d54eb43-3eb3-46d7-b52d-ab15cf88e7d4" containerName="revoke-keys"
Jul 17 19:12:55.421746 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:55.421618    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea755110-0a74-4fcf-9468-9787abf9f434" containerName="revoke-keys"
Jul 17 19:12:55.425873 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:55.425856    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg"
Jul 17 19:12:55.428897 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:55.428866    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:12:55.431420 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:55.431398    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg"]
Jul 17 19:12:55.513447 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:55.513398    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8be06e60-5fd5-4cd9-8d96-6bf6685b017a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg\" (UID: \"8be06e60-5fd5-4cd9-8d96-6bf6685b017a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg"
Jul 17 19:12:55.614492 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:55.614461    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8be06e60-5fd5-4cd9-8d96-6bf6685b017a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg\" (UID: \"8be06e60-5fd5-4cd9-8d96-6bf6685b017a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg"
Jul 17 19:12:55.615023 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:55.615006    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8be06e60-5fd5-4cd9-8d96-6bf6685b017a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg\" (UID: \"8be06e60-5fd5-4cd9-8d96-6bf6685b017a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg"
Jul 17 19:12:55.737644 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:55.737565    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg"
Jul 17 19:12:55.897411 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:55.897333    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg"]
Jul 17 19:12:55.900250 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:12:55.900225    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be06e60_5fd5_4cd9_8d96_6bf6685b017a.slice/crio-36454eaaf0213dd6042346ddbd385d422c008f41356a070c351354c9e60eec0a WatchSource:0}: Error finding container 36454eaaf0213dd6042346ddbd385d422c008f41356a070c351354c9e60eec0a: Status 404 returned error can't find the container with id 36454eaaf0213dd6042346ddbd385d422c008f41356a070c351354c9e60eec0a
Jul 17 19:12:56.704054 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:56.704012    2559 generic.go:356] "Generic (PLEG): container finished" podID="8be06e60-5fd5-4cd9-8d96-6bf6685b017a" containerID="568a12abc0d4187112602e5509d4bded261e620deabf8840723b4d3bc5e5e230" exitCode=6
Jul 17 19:12:56.704408 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:56.704100    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg" event={"ID":"8be06e60-5fd5-4cd9-8d96-6bf6685b017a","Type":"ContainerDied","Data":"568a12abc0d4187112602e5509d4bded261e620deabf8840723b4d3bc5e5e230"}
Jul 17 19:12:56.704408 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:56.704157    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg" event={"ID":"8be06e60-5fd5-4cd9-8d96-6bf6685b017a","Type":"ContainerStarted","Data":"36454eaaf0213dd6042346ddbd385d422c008f41356a070c351354c9e60eec0a"}
Jul 17 19:12:56.704408 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:56.704248    2559 scope.go:117] "RemoveContainer" containerID="568a12abc0d4187112602e5509d4bded261e620deabf8840723b4d3bc5e5e230"
Jul 17 19:12:57.712412 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:57.712296    2559 generic.go:356] "Generic (PLEG): container finished" podID="8be06e60-5fd5-4cd9-8d96-6bf6685b017a" containerID="2ec9ed36bfe802afb270602c0546afc5bb3940680f095cd006d99b7f3b55816a" exitCode=6
Jul 17 19:12:57.712412 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:57.712345    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg" event={"ID":"8be06e60-5fd5-4cd9-8d96-6bf6685b017a","Type":"ContainerDied","Data":"2ec9ed36bfe802afb270602c0546afc5bb3940680f095cd006d99b7f3b55816a"}
Jul 17 19:12:57.712412 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:57.712406    2559 scope.go:117] "RemoveContainer" containerID="568a12abc0d4187112602e5509d4bded261e620deabf8840723b4d3bc5e5e230"
Jul 17 19:12:57.712967 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:57.712662    2559 scope.go:117] "RemoveContainer" containerID="2ec9ed36bfe802afb270602c0546afc5bb3940680f095cd006d99b7f3b55816a"
Jul 17 19:12:57.712967 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:12:57.712905    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg_odh-ai-gateway-infra(8be06e60-5fd5-4cd9-8d96-6bf6685b017a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg" podUID="8be06e60-5fd5-4cd9-8d96-6bf6685b017a"
Jul 17 19:12:58.719768 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:12:58.719740    2559 scope.go:117] "RemoveContainer" containerID="2ec9ed36bfe802afb270602c0546afc5bb3940680f095cd006d99b7f3b55816a"
Jul 17 19:12:58.720187 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:12:58.719927    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg_odh-ai-gateway-infra(8be06e60-5fd5-4cd9-8d96-6bf6685b017a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg" podUID="8be06e60-5fd5-4cd9-8d96-6bf6685b017a"
Jul 17 19:13:10.313543 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:10.313518    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 19:13:10.320629 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:10.320600    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 19:13:10.321210 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:10.321194    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 19:13:10.328002 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:10.327984    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 19:13:12.480263 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:12.480227    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6"]
Jul 17 19:13:12.480772 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:12.480631    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea755110-0a74-4fcf-9468-9787abf9f434" containerName="revoke-keys"
Jul 17 19:13:12.480772 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:12.480643    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea755110-0a74-4fcf-9468-9787abf9f434" containerName="revoke-keys"
Jul 17 19:13:12.480772 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:12.480717    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea755110-0a74-4fcf-9468-9787abf9f434" containerName="revoke-keys"
Jul 17 19:13:12.485173 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:12.485156    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6"
Jul 17 19:13:12.491594 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:12.491562    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6"]
Jul 17 19:13:12.542213 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:12.542185    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1d837e-5fd3-43ea-acc9-8b6755d899fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6\" (UID: \"5b1d837e-5fd3-43ea-acc9-8b6755d899fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6"
Jul 17 19:13:12.642740 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:12.642711    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1d837e-5fd3-43ea-acc9-8b6755d899fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6\" (UID: \"5b1d837e-5fd3-43ea-acc9-8b6755d899fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6"
Jul 17 19:13:12.643254 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:12.643238    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1d837e-5fd3-43ea-acc9-8b6755d899fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6\" (UID: \"5b1d837e-5fd3-43ea-acc9-8b6755d899fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6"
Jul 17 19:13:12.797468 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:12.797391    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6"
Jul 17 19:13:12.937607 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:12.937581    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6"]
Jul 17 19:13:12.939047 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:13:12.939020    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1d837e_5fd3_43ea_acc9_8b6755d899fe.slice/crio-ab21aca6bdcc2d807d18c4b53ce8a52937b4c340a24331a471321bd9d311ef25 WatchSource:0}: Error finding container ab21aca6bdcc2d807d18c4b53ce8a52937b4c340a24331a471321bd9d311ef25: Status 404 returned error can't find the container with id ab21aca6bdcc2d807d18c4b53ce8a52937b4c340a24331a471321bd9d311ef25
Jul 17 19:13:13.804464 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:13.804382    2559 generic.go:356] "Generic (PLEG): container finished" podID="5b1d837e-5fd3-43ea-acc9-8b6755d899fe" containerID="5f4d7bef9ed0c9b5ae6285bc594cc684985bc3405cb2b3092f0037a4b91ef393" exitCode=7
Jul 17 19:13:13.804812 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:13.804482    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6" event={"ID":"5b1d837e-5fd3-43ea-acc9-8b6755d899fe","Type":"ContainerDied","Data":"5f4d7bef9ed0c9b5ae6285bc594cc684985bc3405cb2b3092f0037a4b91ef393"}
Jul 17 19:13:13.804812 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:13.804541    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6" event={"ID":"5b1d837e-5fd3-43ea-acc9-8b6755d899fe","Type":"ContainerStarted","Data":"ab21aca6bdcc2d807d18c4b53ce8a52937b4c340a24331a471321bd9d311ef25"}
Jul 17 19:13:13.804812 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:13.804644    2559 scope.go:117] "RemoveContainer" containerID="5f4d7bef9ed0c9b5ae6285bc594cc684985bc3405cb2b3092f0037a4b91ef393"
Jul 17 19:13:14.120281 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:14.120249    2559 scope.go:117] "RemoveContainer" containerID="2ec9ed36bfe802afb270602c0546afc5bb3940680f095cd006d99b7f3b55816a"
Jul 17 19:13:14.812936 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:14.812897    2559 generic.go:356] "Generic (PLEG): container finished" podID="5b1d837e-5fd3-43ea-acc9-8b6755d899fe" containerID="770d95c07b2ea422f32a0cef0c0442d5c0a0ea418cab29d5846afd5f2458f189" exitCode=7
Jul 17 19:13:14.813334 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:14.812975    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6" event={"ID":"5b1d837e-5fd3-43ea-acc9-8b6755d899fe","Type":"ContainerDied","Data":"770d95c07b2ea422f32a0cef0c0442d5c0a0ea418cab29d5846afd5f2458f189"}
Jul 17 19:13:14.813334 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:14.813038    2559 scope.go:117] "RemoveContainer" containerID="5f4d7bef9ed0c9b5ae6285bc594cc684985bc3405cb2b3092f0037a4b91ef393"
Jul 17 19:13:14.813334 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:14.813198    2559 scope.go:117] "RemoveContainer" containerID="770d95c07b2ea422f32a0cef0c0442d5c0a0ea418cab29d5846afd5f2458f189"
Jul 17 19:13:14.813487 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:13:14.813439    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6_odh-ai-gateway-infra(5b1d837e-5fd3-43ea-acc9-8b6755d899fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6" podUID="5b1d837e-5fd3-43ea-acc9-8b6755d899fe"
Jul 17 19:13:14.815045 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:14.815025    2559 generic.go:356] "Generic (PLEG): container finished" podID="8be06e60-5fd5-4cd9-8d96-6bf6685b017a" containerID="3454bfee44cc20a77094d8754c22d2d1301d90da324705979cb01c2084c068b4" exitCode=6
Jul 17 19:13:14.815142 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:14.815056    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg" event={"ID":"8be06e60-5fd5-4cd9-8d96-6bf6685b017a","Type":"ContainerDied","Data":"3454bfee44cc20a77094d8754c22d2d1301d90da324705979cb01c2084c068b4"}
Jul 17 19:13:14.815333 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:14.815298    2559 scope.go:117] "RemoveContainer" containerID="3454bfee44cc20a77094d8754c22d2d1301d90da324705979cb01c2084c068b4"
Jul 17 19:13:14.815512 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:13:14.815497    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg_odh-ai-gateway-infra(8be06e60-5fd5-4cd9-8d96-6bf6685b017a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg" podUID="8be06e60-5fd5-4cd9-8d96-6bf6685b017a"
Jul 17 19:13:14.824475 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:14.824458    2559 scope.go:117] "RemoveContainer" containerID="2ec9ed36bfe802afb270602c0546afc5bb3940680f095cd006d99b7f3b55816a"
Jul 17 19:13:15.822882 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:15.822857    2559 scope.go:117] "RemoveContainer" containerID="770d95c07b2ea422f32a0cef0c0442d5c0a0ea418cab29d5846afd5f2458f189"
Jul 17 19:13:15.823345 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:13:15.823074    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6_odh-ai-gateway-infra(5b1d837e-5fd3-43ea-acc9-8b6755d899fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6" podUID="5b1d837e-5fd3-43ea-acc9-8b6755d899fe"
Jul 17 19:13:15.856728 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:15.856697    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg"]
Jul 17 19:13:15.993583 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:15.993557    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg"
Jul 17 19:13:16.075853 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:16.075775    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8be06e60-5fd5-4cd9-8d96-6bf6685b017a-maas-api-service-ca\") pod \"8be06e60-5fd5-4cd9-8d96-6bf6685b017a\" (UID: \"8be06e60-5fd5-4cd9-8d96-6bf6685b017a\") "
Jul 17 19:13:16.076160 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:16.076136    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be06e60-5fd5-4cd9-8d96-6bf6685b017a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8be06e60-5fd5-4cd9-8d96-6bf6685b017a" (UID: "8be06e60-5fd5-4cd9-8d96-6bf6685b017a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:13:16.176809 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:16.176771    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8be06e60-5fd5-4cd9-8d96-6bf6685b017a-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:13:16.831034 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:16.831003    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg"
Jul 17 19:13:16.831034 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:16.831014    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg" event={"ID":"8be06e60-5fd5-4cd9-8d96-6bf6685b017a","Type":"ContainerDied","Data":"36454eaaf0213dd6042346ddbd385d422c008f41356a070c351354c9e60eec0a"}
Jul 17 19:13:16.831518 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:16.831077    2559 scope.go:117] "RemoveContainer" containerID="3454bfee44cc20a77094d8754c22d2d1301d90da324705979cb01c2084c068b4"
Jul 17 19:13:16.850764 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:16.850738    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg"]
Jul 17 19:13:16.854873 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:16.854852    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-85vhg"]
Jul 17 19:13:18.125481 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:18.125441    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8be06e60-5fd5-4cd9-8d96-6bf6685b017a" path="/var/lib/kubelet/pods/8be06e60-5fd5-4cd9-8d96-6bf6685b017a/volumes"
Jul 17 19:13:30.122620 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:30.122538    2559 scope.go:117] "RemoveContainer" containerID="770d95c07b2ea422f32a0cef0c0442d5c0a0ea418cab29d5846afd5f2458f189"
Jul 17 19:13:30.915375 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:30.915342    2559 generic.go:356] "Generic (PLEG): container finished" podID="5b1d837e-5fd3-43ea-acc9-8b6755d899fe" containerID="979d4524d06ca7b5114852952593dadeb8f8c585422a8faad4803b6e9c78b237" exitCode=6
Jul 17 19:13:30.915469 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:30.915416    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6" event={"ID":"5b1d837e-5fd3-43ea-acc9-8b6755d899fe","Type":"ContainerDied","Data":"979d4524d06ca7b5114852952593dadeb8f8c585422a8faad4803b6e9c78b237"}
Jul 17 19:13:30.915519 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:30.915502    2559 scope.go:117] "RemoveContainer" containerID="770d95c07b2ea422f32a0cef0c0442d5c0a0ea418cab29d5846afd5f2458f189"
Jul 17 19:13:30.915758 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:30.915741    2559 scope.go:117] "RemoveContainer" containerID="979d4524d06ca7b5114852952593dadeb8f8c585422a8faad4803b6e9c78b237"
Jul 17 19:13:30.915955 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:13:30.915930    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6_odh-ai-gateway-infra(5b1d837e-5fd3-43ea-acc9-8b6755d899fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6" podUID="5b1d837e-5fd3-43ea-acc9-8b6755d899fe"
Jul 17 19:13:31.943452 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:31.943417    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6"]
Jul 17 19:13:32.076202 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:32.076182    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6"
Jul 17 19:13:32.215724 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:32.215654    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1d837e-5fd3-43ea-acc9-8b6755d899fe-maas-api-service-ca\") pod \"5b1d837e-5fd3-43ea-acc9-8b6755d899fe\" (UID: \"5b1d837e-5fd3-43ea-acc9-8b6755d899fe\") "
Jul 17 19:13:32.215961 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:32.215941    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b1d837e-5fd3-43ea-acc9-8b6755d899fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b1d837e-5fd3-43ea-acc9-8b6755d899fe" (UID: "5b1d837e-5fd3-43ea-acc9-8b6755d899fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:13:32.317073 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:32.317043    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1d837e-5fd3-43ea-acc9-8b6755d899fe-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:13:32.929864 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:32.929830    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6" event={"ID":"5b1d837e-5fd3-43ea-acc9-8b6755d899fe","Type":"ContainerDied","Data":"ab21aca6bdcc2d807d18c4b53ce8a52937b4c340a24331a471321bd9d311ef25"}
Jul 17 19:13:32.930045 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:32.929896    2559 scope.go:117] "RemoveContainer" containerID="979d4524d06ca7b5114852952593dadeb8f8c585422a8faad4803b6e9c78b237"
Jul 17 19:13:32.930045 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:32.929846    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6"
Jul 17 19:13:32.953017 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:32.952984    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6"]
Jul 17 19:13:32.958257 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:32.958234    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-t2pv6"]
Jul 17 19:13:34.125913 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:34.125879    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b1d837e-5fd3-43ea-acc9-8b6755d899fe" path="/var/lib/kubelet/pods/5b1d837e-5fd3-43ea-acc9-8b6755d899fe/volumes"
Jul 17 19:13:40.796330 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:40.796275    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj"]
Jul 17 19:13:40.796715 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:40.796705    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be06e60-5fd5-4cd9-8d96-6bf6685b017a" containerName="revoke-keys"
Jul 17 19:13:40.796755 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:40.796718    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be06e60-5fd5-4cd9-8d96-6bf6685b017a" containerName="revoke-keys"
Jul 17 19:13:40.796755 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:40.796727    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be06e60-5fd5-4cd9-8d96-6bf6685b017a" containerName="revoke-keys"
Jul 17 19:13:40.796755 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:40.796733    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be06e60-5fd5-4cd9-8d96-6bf6685b017a" containerName="revoke-keys"
Jul 17 19:13:40.796755 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:40.796739    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b1d837e-5fd3-43ea-acc9-8b6755d899fe" containerName="revoke-keys"
Jul 17 19:13:40.796755 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:40.796744    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b1d837e-5fd3-43ea-acc9-8b6755d899fe" containerName="revoke-keys"
Jul 17 19:13:40.796908 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:40.796762    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b1d837e-5fd3-43ea-acc9-8b6755d899fe" containerName="revoke-keys"
Jul 17 19:13:40.796908 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:40.796767    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b1d837e-5fd3-43ea-acc9-8b6755d899fe" containerName="revoke-keys"
Jul 17 19:13:40.796908 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:40.796823    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b1d837e-5fd3-43ea-acc9-8b6755d899fe" containerName="revoke-keys"
Jul 17 19:13:40.796908 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:40.796830    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8be06e60-5fd5-4cd9-8d96-6bf6685b017a" containerName="revoke-keys"
Jul 17 19:13:40.796908 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:40.796838    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8be06e60-5fd5-4cd9-8d96-6bf6685b017a" containerName="revoke-keys"
Jul 17 19:13:40.796908 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:40.796847    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b1d837e-5fd3-43ea-acc9-8b6755d899fe" containerName="revoke-keys"
Jul 17 19:13:40.796908 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:40.796852    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8be06e60-5fd5-4cd9-8d96-6bf6685b017a" containerName="revoke-keys"
Jul 17 19:13:40.801234 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:40.801216    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj"
Jul 17 19:13:40.803934 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:40.803909    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:13:40.806187 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:40.806166    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj"]
Jul 17 19:13:40.897573 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:40.897539    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e86a21d-7850-4be9-ab6b-d710fec86319-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj\" (UID: \"1e86a21d-7850-4be9-ab6b-d710fec86319\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj"
Jul 17 19:13:40.998072 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:40.998042    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e86a21d-7850-4be9-ab6b-d710fec86319-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj\" (UID: \"1e86a21d-7850-4be9-ab6b-d710fec86319\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj"
Jul 17 19:13:40.998671 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:40.998652    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e86a21d-7850-4be9-ab6b-d710fec86319-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj\" (UID: \"1e86a21d-7850-4be9-ab6b-d710fec86319\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj"
Jul 17 19:13:41.113064 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:41.113037    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj"
Jul 17 19:13:41.254570 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:41.254544    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj"]
Jul 17 19:13:41.256042 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:13:41.256011    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e86a21d_7850_4be9_ab6b_d710fec86319.slice/crio-b900ba5923ffa4d967b78653dc59fa58a4592f8166d7051e955f80fb879db037 WatchSource:0}: Error finding container b900ba5923ffa4d967b78653dc59fa58a4592f8166d7051e955f80fb879db037: Status 404 returned error can't find the container with id b900ba5923ffa4d967b78653dc59fa58a4592f8166d7051e955f80fb879db037
Jul 17 19:13:41.981366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:41.981340    2559 generic.go:356] "Generic (PLEG): container finished" podID="1e86a21d-7850-4be9-ab6b-d710fec86319" containerID="7f70f9460c7d1c866a88bdf0ab59e97ed1d89f0468cdcaf4376304c760f839ed" exitCode=7
Jul 17 19:13:41.981681 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:41.981436    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj" event={"ID":"1e86a21d-7850-4be9-ab6b-d710fec86319","Type":"ContainerDied","Data":"7f70f9460c7d1c866a88bdf0ab59e97ed1d89f0468cdcaf4376304c760f839ed"}
Jul 17 19:13:41.981681 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:41.981494    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj" event={"ID":"1e86a21d-7850-4be9-ab6b-d710fec86319","Type":"ContainerStarted","Data":"b900ba5923ffa4d967b78653dc59fa58a4592f8166d7051e955f80fb879db037"}
Jul 17 19:13:41.981681 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:41.981574    2559 scope.go:117] "RemoveContainer" containerID="7f70f9460c7d1c866a88bdf0ab59e97ed1d89f0468cdcaf4376304c760f839ed"
Jul 17 19:13:42.989711 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:42.989670    2559 generic.go:356] "Generic (PLEG): container finished" podID="1e86a21d-7850-4be9-ab6b-d710fec86319" containerID="30d0cf3aa312138162ba256cfadb1876dde3b6e6f2c968e46497dedca1216216" exitCode=7
Jul 17 19:13:42.990067 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:42.989758    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj" event={"ID":"1e86a21d-7850-4be9-ab6b-d710fec86319","Type":"ContainerDied","Data":"30d0cf3aa312138162ba256cfadb1876dde3b6e6f2c968e46497dedca1216216"}
Jul 17 19:13:42.990067 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:42.989858    2559 scope.go:117] "RemoveContainer" containerID="7f70f9460c7d1c866a88bdf0ab59e97ed1d89f0468cdcaf4376304c760f839ed"
Jul 17 19:13:42.990152 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:42.990100    2559 scope.go:117] "RemoveContainer" containerID="30d0cf3aa312138162ba256cfadb1876dde3b6e6f2c968e46497dedca1216216"
Jul 17 19:13:42.990350 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:13:42.990297    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj_odh-ai-gateway-infra(1e86a21d-7850-4be9-ab6b-d710fec86319)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj" podUID="1e86a21d-7850-4be9-ab6b-d710fec86319"
Jul 17 19:13:43.996572 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:43.996540    2559 scope.go:117] "RemoveContainer" containerID="30d0cf3aa312138162ba256cfadb1876dde3b6e6f2c968e46497dedca1216216"
Jul 17 19:13:43.996944 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:13:43.996714    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj_odh-ai-gateway-infra(1e86a21d-7850-4be9-ab6b-d710fec86319)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj" podUID="1e86a21d-7850-4be9-ab6b-d710fec86319"
Jul 17 19:13:55.477013 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:55.476972    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x"]
Jul 17 19:13:55.477613 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:55.477591    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be06e60-5fd5-4cd9-8d96-6bf6685b017a" containerName="revoke-keys"
Jul 17 19:13:55.477688 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:55.477614    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be06e60-5fd5-4cd9-8d96-6bf6685b017a" containerName="revoke-keys"
Jul 17 19:13:55.477688 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:55.477627    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b1d837e-5fd3-43ea-acc9-8b6755d899fe" containerName="revoke-keys"
Jul 17 19:13:55.477688 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:55.477634    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b1d837e-5fd3-43ea-acc9-8b6755d899fe" containerName="revoke-keys"
Jul 17 19:13:55.477846 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:55.477746    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b1d837e-5fd3-43ea-acc9-8b6755d899fe" containerName="revoke-keys"
Jul 17 19:13:55.480829 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:55.480808    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x"
Jul 17 19:13:55.488511 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:55.488237    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x"]
Jul 17 19:13:55.518464 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:55.518434    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf025313-af41-4dd3-a352-87e016be4726-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x\" (UID: \"bf025313-af41-4dd3-a352-87e016be4726\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x"
Jul 17 19:13:55.619185 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:55.619156    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf025313-af41-4dd3-a352-87e016be4726-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x\" (UID: \"bf025313-af41-4dd3-a352-87e016be4726\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x"
Jul 17 19:13:55.619836 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:55.619817    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf025313-af41-4dd3-a352-87e016be4726-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x\" (UID: \"bf025313-af41-4dd3-a352-87e016be4726\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x"
Jul 17 19:13:55.793911 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:55.793839    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x"
Jul 17 19:13:55.945684 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:55.945661    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x"]
Jul 17 19:13:55.948476 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:13:55.948445    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf025313_af41_4dd3_a352_87e016be4726.slice/crio-44a1da6c1bd9bcd1795fbac2a2c95403fe1f2a5d429066d3f42bca1fb83304e4 WatchSource:0}: Error finding container 44a1da6c1bd9bcd1795fbac2a2c95403fe1f2a5d429066d3f42bca1fb83304e4: Status 404 returned error can't find the container with id 44a1da6c1bd9bcd1795fbac2a2c95403fe1f2a5d429066d3f42bca1fb83304e4
Jul 17 19:13:56.064099 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:56.064071    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x" event={"ID":"bf025313-af41-4dd3-a352-87e016be4726","Type":"ContainerStarted","Data":"44a1da6c1bd9bcd1795fbac2a2c95403fe1f2a5d429066d3f42bca1fb83304e4"}
Jul 17 19:13:57.070971 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:57.070939    2559 generic.go:356] "Generic (PLEG): container finished" podID="bf025313-af41-4dd3-a352-87e016be4726" containerID="9b4ef30d729fb48ebe31670787a219f00e617928e7e5c8ab94ee01a3f9f09738" exitCode=6
Jul 17 19:13:57.071378 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:57.071017    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x" event={"ID":"bf025313-af41-4dd3-a352-87e016be4726","Type":"ContainerDied","Data":"9b4ef30d729fb48ebe31670787a219f00e617928e7e5c8ab94ee01a3f9f09738"}
Jul 17 19:13:57.071378 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:57.071168    2559 scope.go:117] "RemoveContainer" containerID="9b4ef30d729fb48ebe31670787a219f00e617928e7e5c8ab94ee01a3f9f09738"
Jul 17 19:13:58.079163 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:58.079123    2559 generic.go:356] "Generic (PLEG): container finished" podID="bf025313-af41-4dd3-a352-87e016be4726" containerID="b64d52a82decbddeb3f47ddc960ef6ac35c444780f6453760851aa463690388d" exitCode=6
Jul 17 19:13:58.079602 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:58.079172    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x" event={"ID":"bf025313-af41-4dd3-a352-87e016be4726","Type":"ContainerDied","Data":"b64d52a82decbddeb3f47ddc960ef6ac35c444780f6453760851aa463690388d"}
Jul 17 19:13:58.079602 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:58.079220    2559 scope.go:117] "RemoveContainer" containerID="9b4ef30d729fb48ebe31670787a219f00e617928e7e5c8ab94ee01a3f9f09738"
Jul 17 19:13:58.079602 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:58.079409    2559 scope.go:117] "RemoveContainer" containerID="b64d52a82decbddeb3f47ddc960ef6ac35c444780f6453760851aa463690388d"
Jul 17 19:13:58.079727 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:13:58.079608    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x_odh-ai-gateway-infra(bf025313-af41-4dd3-a352-87e016be4726)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x" podUID="bf025313-af41-4dd3-a352-87e016be4726"
Jul 17 19:13:58.119998 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:58.119979    2559 scope.go:117] "RemoveContainer" containerID="30d0cf3aa312138162ba256cfadb1876dde3b6e6f2c968e46497dedca1216216"
Jul 17 19:13:59.086008 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:59.085977    2559 generic.go:356] "Generic (PLEG): container finished" podID="1e86a21d-7850-4be9-ab6b-d710fec86319" containerID="4adcadd052d9b2f2871963c3af22e3ad8428ac74d165e913b4a4d21c411d83df" exitCode=7
Jul 17 19:13:59.086473 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:59.086049    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj" event={"ID":"1e86a21d-7850-4be9-ab6b-d710fec86319","Type":"ContainerDied","Data":"4adcadd052d9b2f2871963c3af22e3ad8428ac74d165e913b4a4d21c411d83df"}
Jul 17 19:13:59.086473 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:59.086105    2559 scope.go:117] "RemoveContainer" containerID="30d0cf3aa312138162ba256cfadb1876dde3b6e6f2c968e46497dedca1216216"
Jul 17 19:13:59.086473 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:59.086457    2559 scope.go:117] "RemoveContainer" containerID="4adcadd052d9b2f2871963c3af22e3ad8428ac74d165e913b4a4d21c411d83df"
Jul 17 19:13:59.086704 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:13:59.086681    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj_odh-ai-gateway-infra(1e86a21d-7850-4be9-ab6b-d710fec86319)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj" podUID="1e86a21d-7850-4be9-ab6b-d710fec86319"
Jul 17 19:13:59.087946 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:13:59.087930    2559 scope.go:117] "RemoveContainer" containerID="b64d52a82decbddeb3f47ddc960ef6ac35c444780f6453760851aa463690388d"
Jul 17 19:13:59.088114 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:13:59.088098    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x_odh-ai-gateway-infra(bf025313-af41-4dd3-a352-87e016be4726)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x" podUID="bf025313-af41-4dd3-a352-87e016be4726"
Jul 17 19:14:00.113744 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:00.113711    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj"]
Jul 17 19:14:00.254017 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:00.253993    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj"
Jul 17 19:14:00.360089 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:00.360062    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e86a21d-7850-4be9-ab6b-d710fec86319-maas-api-service-ca\") pod \"1e86a21d-7850-4be9-ab6b-d710fec86319\" (UID: \"1e86a21d-7850-4be9-ab6b-d710fec86319\") "
Jul 17 19:14:00.360420 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:00.360398    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e86a21d-7850-4be9-ab6b-d710fec86319-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1e86a21d-7850-4be9-ab6b-d710fec86319" (UID: "1e86a21d-7850-4be9-ab6b-d710fec86319"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:14:00.461047 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:00.460988    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e86a21d-7850-4be9-ab6b-d710fec86319-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:14:01.102811 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:01.102786    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj"
Jul 17 19:14:01.102963 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:01.102824    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj" event={"ID":"1e86a21d-7850-4be9-ab6b-d710fec86319","Type":"ContainerDied","Data":"b900ba5923ffa4d967b78653dc59fa58a4592f8166d7051e955f80fb879db037"}
Jul 17 19:14:01.102963 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:01.102910    2559 scope.go:117] "RemoveContainer" containerID="4adcadd052d9b2f2871963c3af22e3ad8428ac74d165e913b4a4d21c411d83df"
Jul 17 19:14:01.127440 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:01.127410    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj"]
Jul 17 19:14:01.132962 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:01.132939    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q7dvj"]
Jul 17 19:14:02.125300 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:02.125270    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e86a21d-7850-4be9-ab6b-d710fec86319" path="/var/lib/kubelet/pods/1e86a21d-7850-4be9-ab6b-d710fec86319/volumes"
Jul 17 19:14:12.541266 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:12.541227    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v"]
Jul 17 19:14:12.541769 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:12.541624    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e86a21d-7850-4be9-ab6b-d710fec86319" containerName="revoke-keys"
Jul 17 19:14:12.541769 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:12.541636    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e86a21d-7850-4be9-ab6b-d710fec86319" containerName="revoke-keys"
Jul 17 19:14:12.541769 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:12.541658    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e86a21d-7850-4be9-ab6b-d710fec86319" containerName="revoke-keys"
Jul 17 19:14:12.541769 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:12.541663    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e86a21d-7850-4be9-ab6b-d710fec86319" containerName="revoke-keys"
Jul 17 19:14:12.541769 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:12.541723    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e86a21d-7850-4be9-ab6b-d710fec86319" containerName="revoke-keys"
Jul 17 19:14:12.541769 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:12.541732    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e86a21d-7850-4be9-ab6b-d710fec86319" containerName="revoke-keys"
Jul 17 19:14:12.546057 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:12.546041    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v"
Jul 17 19:14:12.553806 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:12.553374    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v"]
Jul 17 19:14:12.666541 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:12.666506    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af3d00a-2e9e-4f07-9eae-d518ffc5c19e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v\" (UID: \"6af3d00a-2e9e-4f07-9eae-d518ffc5c19e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v"
Jul 17 19:14:12.767614 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:12.767576    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af3d00a-2e9e-4f07-9eae-d518ffc5c19e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v\" (UID: \"6af3d00a-2e9e-4f07-9eae-d518ffc5c19e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v"
Jul 17 19:14:12.768187 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:12.768171    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af3d00a-2e9e-4f07-9eae-d518ffc5c19e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v\" (UID: \"6af3d00a-2e9e-4f07-9eae-d518ffc5c19e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v"
Jul 17 19:14:12.857806 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:12.857784    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v"
Jul 17 19:14:12.992328 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:12.992287    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v"]
Jul 17 19:14:12.993735 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:14:12.993709    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af3d00a_2e9e_4f07_9eae_d518ffc5c19e.slice/crio-0f32a57f6a8a842c2fcaf98a01b5fcebd72b9c60b5c24179e0bd3b52db28849a WatchSource:0}: Error finding container 0f32a57f6a8a842c2fcaf98a01b5fcebd72b9c60b5c24179e0bd3b52db28849a: Status 404 returned error can't find the container with id 0f32a57f6a8a842c2fcaf98a01b5fcebd72b9c60b5c24179e0bd3b52db28849a
Jul 17 19:14:13.172719 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:13.172652    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v" event={"ID":"6af3d00a-2e9e-4f07-9eae-d518ffc5c19e","Type":"ContainerStarted","Data":"0f32a57f6a8a842c2fcaf98a01b5fcebd72b9c60b5c24179e0bd3b52db28849a"}
Jul 17 19:14:14.120007 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:14.119982    2559 scope.go:117] "RemoveContainer" containerID="b64d52a82decbddeb3f47ddc960ef6ac35c444780f6453760851aa463690388d"
Jul 17 19:14:14.179595 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:14.179570    2559 generic.go:356] "Generic (PLEG): container finished" podID="6af3d00a-2e9e-4f07-9eae-d518ffc5c19e" containerID="52f428ed48b72870f1ed5c178f473d42fabca7567e0e78e3c423b3aa1fa302e0" exitCode=6
Jul 17 19:14:14.179706 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:14.179665    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v" event={"ID":"6af3d00a-2e9e-4f07-9eae-d518ffc5c19e","Type":"ContainerDied","Data":"52f428ed48b72870f1ed5c178f473d42fabca7567e0e78e3c423b3aa1fa302e0"}
Jul 17 19:14:14.179833 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:14.179820    2559 scope.go:117] "RemoveContainer" containerID="52f428ed48b72870f1ed5c178f473d42fabca7567e0e78e3c423b3aa1fa302e0"
Jul 17 19:14:15.187151 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:15.187122    2559 generic.go:356] "Generic (PLEG): container finished" podID="6af3d00a-2e9e-4f07-9eae-d518ffc5c19e" containerID="97f37e8f1db27fb133140861a1b07975245be0b9f1b2c92f974a0000821ccba1" exitCode=6
Jul 17 19:14:15.187581 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:15.187208    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v" event={"ID":"6af3d00a-2e9e-4f07-9eae-d518ffc5c19e","Type":"ContainerDied","Data":"97f37e8f1db27fb133140861a1b07975245be0b9f1b2c92f974a0000821ccba1"}
Jul 17 19:14:15.187581 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:15.187267    2559 scope.go:117] "RemoveContainer" containerID="52f428ed48b72870f1ed5c178f473d42fabca7567e0e78e3c423b3aa1fa302e0"
Jul 17 19:14:15.187581 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:15.187500    2559 scope.go:117] "RemoveContainer" containerID="97f37e8f1db27fb133140861a1b07975245be0b9f1b2c92f974a0000821ccba1"
Jul 17 19:14:15.187727 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:14:15.187709    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v_odh-ai-gateway-infra(6af3d00a-2e9e-4f07-9eae-d518ffc5c19e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v" podUID="6af3d00a-2e9e-4f07-9eae-d518ffc5c19e"
Jul 17 19:14:15.189340 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:15.189293    2559 generic.go:356] "Generic (PLEG): container finished" podID="bf025313-af41-4dd3-a352-87e016be4726" containerID="8da334f5ffe5b31c1bee54e83242f494c1899bc7913d6562a5266c7eb14dadc6" exitCode=6
Jul 17 19:14:15.189452 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:15.189336    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x" event={"ID":"bf025313-af41-4dd3-a352-87e016be4726","Type":"ContainerDied","Data":"8da334f5ffe5b31c1bee54e83242f494c1899bc7913d6562a5266c7eb14dadc6"}
Jul 17 19:14:15.189621 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:15.189607    2559 scope.go:117] "RemoveContainer" containerID="8da334f5ffe5b31c1bee54e83242f494c1899bc7913d6562a5266c7eb14dadc6"
Jul 17 19:14:15.189762 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:14:15.189746    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x_odh-ai-gateway-infra(bf025313-af41-4dd3-a352-87e016be4726)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x" podUID="bf025313-af41-4dd3-a352-87e016be4726"
Jul 17 19:14:15.201479 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:15.201196    2559 scope.go:117] "RemoveContainer" containerID="b64d52a82decbddeb3f47ddc960ef6ac35c444780f6453760851aa463690388d"
Jul 17 19:14:16.197583 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:16.197556    2559 scope.go:117] "RemoveContainer" containerID="97f37e8f1db27fb133140861a1b07975245be0b9f1b2c92f974a0000821ccba1"
Jul 17 19:14:16.197957 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:14:16.197727    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v_odh-ai-gateway-infra(6af3d00a-2e9e-4f07-9eae-d518ffc5c19e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v" podUID="6af3d00a-2e9e-4f07-9eae-d518ffc5c19e"
Jul 17 19:14:16.226015 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:16.225987    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x"]
Jul 17 19:14:16.353909 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:16.353884    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x"
Jul 17 19:14:16.496556 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:16.496488    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf025313-af41-4dd3-a352-87e016be4726-maas-api-service-ca\") pod \"bf025313-af41-4dd3-a352-87e016be4726\" (UID: \"bf025313-af41-4dd3-a352-87e016be4726\") "
Jul 17 19:14:16.496811 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:16.496790    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf025313-af41-4dd3-a352-87e016be4726-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bf025313-af41-4dd3-a352-87e016be4726" (UID: "bf025313-af41-4dd3-a352-87e016be4726"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:14:16.597624 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:16.597599    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf025313-af41-4dd3-a352-87e016be4726-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:14:17.204423 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:17.204395    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x"
Jul 17 19:14:17.204857 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:17.204427    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x" event={"ID":"bf025313-af41-4dd3-a352-87e016be4726","Type":"ContainerDied","Data":"44a1da6c1bd9bcd1795fbac2a2c95403fe1f2a5d429066d3f42bca1fb83304e4"}
Jul 17 19:14:17.204857 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:17.204478    2559 scope.go:117] "RemoveContainer" containerID="8da334f5ffe5b31c1bee54e83242f494c1899bc7913d6562a5266c7eb14dadc6"
Jul 17 19:14:17.227821 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:17.227796    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x"]
Jul 17 19:14:17.232012 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:17.231991    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-89b5x"]
Jul 17 19:14:18.125646 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:18.125617    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf025313-af41-4dd3-a352-87e016be4726" path="/var/lib/kubelet/pods/bf025313-af41-4dd3-a352-87e016be4726/volumes"
Jul 17 19:14:28.119883 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:28.119848    2559 scope.go:117] "RemoveContainer" containerID="97f37e8f1db27fb133140861a1b07975245be0b9f1b2c92f974a0000821ccba1"
Jul 17 19:14:29.272137 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:29.272104    2559 generic.go:356] "Generic (PLEG): container finished" podID="6af3d00a-2e9e-4f07-9eae-d518ffc5c19e" containerID="58f492b3ba3a99674015cb14769c28861ce2e2ddf77976566aaf04ceb945747f" exitCode=6
Jul 17 19:14:29.272601 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:29.272142    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v" event={"ID":"6af3d00a-2e9e-4f07-9eae-d518ffc5c19e","Type":"ContainerDied","Data":"58f492b3ba3a99674015cb14769c28861ce2e2ddf77976566aaf04ceb945747f"}
Jul 17 19:14:29.272601 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:29.272208    2559 scope.go:117] "RemoveContainer" containerID="97f37e8f1db27fb133140861a1b07975245be0b9f1b2c92f974a0000821ccba1"
Jul 17 19:14:29.272601 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:29.272480    2559 scope.go:117] "RemoveContainer" containerID="58f492b3ba3a99674015cb14769c28861ce2e2ddf77976566aaf04ceb945747f"
Jul 17 19:14:29.272754 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:14:29.272659    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v_odh-ai-gateway-infra(6af3d00a-2e9e-4f07-9eae-d518ffc5c19e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v" podUID="6af3d00a-2e9e-4f07-9eae-d518ffc5c19e"
Jul 17 19:14:30.304863 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:30.304831    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v"]
Jul 17 19:14:30.465702 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:30.465676    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v"
Jul 17 19:14:30.619083 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:30.619055    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af3d00a-2e9e-4f07-9eae-d518ffc5c19e-maas-api-service-ca\") pod \"6af3d00a-2e9e-4f07-9eae-d518ffc5c19e\" (UID: \"6af3d00a-2e9e-4f07-9eae-d518ffc5c19e\") "
Jul 17 19:14:30.619386 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:30.619366    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af3d00a-2e9e-4f07-9eae-d518ffc5c19e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6af3d00a-2e9e-4f07-9eae-d518ffc5c19e" (UID: "6af3d00a-2e9e-4f07-9eae-d518ffc5c19e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:14:30.720370 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:30.720342    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af3d00a-2e9e-4f07-9eae-d518ffc5c19e-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:14:31.286826 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:31.286796    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v"
Jul 17 19:14:31.286995 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:31.286797    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v" event={"ID":"6af3d00a-2e9e-4f07-9eae-d518ffc5c19e","Type":"ContainerDied","Data":"0f32a57f6a8a842c2fcaf98a01b5fcebd72b9c60b5c24179e0bd3b52db28849a"}
Jul 17 19:14:31.286995 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:31.286934    2559 scope.go:117] "RemoveContainer" containerID="58f492b3ba3a99674015cb14769c28861ce2e2ddf77976566aaf04ceb945747f"
Jul 17 19:14:31.308395 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:31.308366    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v"]
Jul 17 19:14:31.315271 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:31.315250    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-vss5v"]
Jul 17 19:14:32.125647 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:32.125614    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6af3d00a-2e9e-4f07-9eae-d518ffc5c19e" path="/var/lib/kubelet/pods/6af3d00a-2e9e-4f07-9eae-d518ffc5c19e/volumes"
Jul 17 19:14:33.690240 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.690211    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h"]
Jul 17 19:14:33.690697 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.690576    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6af3d00a-2e9e-4f07-9eae-d518ffc5c19e" containerName="revoke-keys"
Jul 17 19:14:33.690697 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.690588    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6af3d00a-2e9e-4f07-9eae-d518ffc5c19e" containerName="revoke-keys"
Jul 17 19:14:33.690697 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.690598    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e86a21d-7850-4be9-ab6b-d710fec86319" containerName="revoke-keys"
Jul 17 19:14:33.690697 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.690603    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e86a21d-7850-4be9-ab6b-d710fec86319" containerName="revoke-keys"
Jul 17 19:14:33.690697 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.690616    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf025313-af41-4dd3-a352-87e016be4726" containerName="revoke-keys"
Jul 17 19:14:33.690697 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.690622    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf025313-af41-4dd3-a352-87e016be4726" containerName="revoke-keys"
Jul 17 19:14:33.690697 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.690631    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf025313-af41-4dd3-a352-87e016be4726" containerName="revoke-keys"
Jul 17 19:14:33.690697 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.690637    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf025313-af41-4dd3-a352-87e016be4726" containerName="revoke-keys"
Jul 17 19:14:33.690697 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.690649    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6af3d00a-2e9e-4f07-9eae-d518ffc5c19e" containerName="revoke-keys"
Jul 17 19:14:33.690697 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.690654    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6af3d00a-2e9e-4f07-9eae-d518ffc5c19e" containerName="revoke-keys"
Jul 17 19:14:33.691219 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.690715    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6af3d00a-2e9e-4f07-9eae-d518ffc5c19e" containerName="revoke-keys"
Jul 17 19:14:33.691219 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.690723    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6af3d00a-2e9e-4f07-9eae-d518ffc5c19e" containerName="revoke-keys"
Jul 17 19:14:33.691219 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.690733    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf025313-af41-4dd3-a352-87e016be4726" containerName="revoke-keys"
Jul 17 19:14:33.691219 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.690742    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf025313-af41-4dd3-a352-87e016be4726" containerName="revoke-keys"
Jul 17 19:14:33.691219 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.690751    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf025313-af41-4dd3-a352-87e016be4726" containerName="revoke-keys"
Jul 17 19:14:33.691219 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.690759    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e86a21d-7850-4be9-ab6b-d710fec86319" containerName="revoke-keys"
Jul 17 19:14:33.695109 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.695094    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h"
Jul 17 19:14:33.697516 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.697497    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:14:33.701888 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.701864    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h"]
Jul 17 19:14:33.849609 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.849581    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h\" (UID: \"0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h"
Jul 17 19:14:33.950523 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.950460    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h\" (UID: \"0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h"
Jul 17 19:14:33.951041 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:33.951023    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h\" (UID: \"0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h"
Jul 17 19:14:34.007391 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:34.007370    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h"
Jul 17 19:14:34.144627 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:34.144580    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h"]
Jul 17 19:14:34.150898 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:14:34.150872    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbc9dd6_7ae3_4a49_9aa0_edc5c4ee2492.slice/crio-6ad663050a9906450e54b685d12bded3b2100034aafc0a6fd43a2d1b569fd610 WatchSource:0}: Error finding container 6ad663050a9906450e54b685d12bded3b2100034aafc0a6fd43a2d1b569fd610: Status 404 returned error can't find the container with id 6ad663050a9906450e54b685d12bded3b2100034aafc0a6fd43a2d1b569fd610
Jul 17 19:14:34.305291 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:34.305221    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h" event={"ID":"0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492","Type":"ContainerStarted","Data":"6ad663050a9906450e54b685d12bded3b2100034aafc0a6fd43a2d1b569fd610"}
Jul 17 19:14:35.312326 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:35.312269    2559 generic.go:356] "Generic (PLEG): container finished" podID="0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492" containerID="3a8a13cd095eebe4ed123f8e0332a40f72e573bbc9c37159ccb6dba9c09a8d30" exitCode=6
Jul 17 19:14:35.312731 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:35.312366    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h" event={"ID":"0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492","Type":"ContainerDied","Data":"3a8a13cd095eebe4ed123f8e0332a40f72e573bbc9c37159ccb6dba9c09a8d30"}
Jul 17 19:14:35.312731 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:35.312534    2559 scope.go:117] "RemoveContainer" containerID="3a8a13cd095eebe4ed123f8e0332a40f72e573bbc9c37159ccb6dba9c09a8d30"
Jul 17 19:14:36.319355 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:36.319299    2559 generic.go:356] "Generic (PLEG): container finished" podID="0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492" containerID="b4ff4031879e1fe3e3ef7371ffcf6e23a3b480f612eab7ebb3e1dd03d83b6f26" exitCode=6
Jul 17 19:14:36.319749 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:36.319423    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h" event={"ID":"0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492","Type":"ContainerDied","Data":"b4ff4031879e1fe3e3ef7371ffcf6e23a3b480f612eab7ebb3e1dd03d83b6f26"}
Jul 17 19:14:36.319749 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:36.319481    2559 scope.go:117] "RemoveContainer" containerID="3a8a13cd095eebe4ed123f8e0332a40f72e573bbc9c37159ccb6dba9c09a8d30"
Jul 17 19:14:36.319749 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:36.319554    2559 scope.go:117] "RemoveContainer" containerID="b4ff4031879e1fe3e3ef7371ffcf6e23a3b480f612eab7ebb3e1dd03d83b6f26"
Jul 17 19:14:36.319749 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:14:36.319739    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h_odh-ai-gateway-infra(0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h" podUID="0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492"
Jul 17 19:14:37.327467 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:37.327438    2559 scope.go:117] "RemoveContainer" containerID="b4ff4031879e1fe3e3ef7371ffcf6e23a3b480f612eab7ebb3e1dd03d83b6f26"
Jul 17 19:14:37.327857 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:14:37.327608    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h_odh-ai-gateway-infra(0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h" podUID="0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492"
Jul 17 19:14:40.840928 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:40.840892    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw"]
Jul 17 19:14:40.841430 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:40.841345    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6af3d00a-2e9e-4f07-9eae-d518ffc5c19e" containerName="revoke-keys"
Jul 17 19:14:40.841430 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:40.841359    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6af3d00a-2e9e-4f07-9eae-d518ffc5c19e" containerName="revoke-keys"
Jul 17 19:14:40.841430 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:40.841384    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf025313-af41-4dd3-a352-87e016be4726" containerName="revoke-keys"
Jul 17 19:14:40.841430 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:40.841391    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf025313-af41-4dd3-a352-87e016be4726" containerName="revoke-keys"
Jul 17 19:14:40.841652 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:40.841463    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6af3d00a-2e9e-4f07-9eae-d518ffc5c19e" containerName="revoke-keys"
Jul 17 19:14:40.845697 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:40.845674    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw"
Jul 17 19:14:40.853590 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:40.853561    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw"]
Jul 17 19:14:40.907007 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:40.906979    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbaaa271-a381-4d42-b517-8d00b4c04db9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw\" (UID: \"cbaaa271-a381-4d42-b517-8d00b4c04db9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw"
Jul 17 19:14:41.008100 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:41.008071    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbaaa271-a381-4d42-b517-8d00b4c04db9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw\" (UID: \"cbaaa271-a381-4d42-b517-8d00b4c04db9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw"
Jul 17 19:14:41.008717 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:41.008699    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbaaa271-a381-4d42-b517-8d00b4c04db9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw\" (UID: \"cbaaa271-a381-4d42-b517-8d00b4c04db9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw"
Jul 17 19:14:41.158850 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:41.158774    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw"
Jul 17 19:14:41.293930 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:41.293902    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw"]
Jul 17 19:14:41.294932 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:14:41.294902    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbaaa271_a381_4d42_b517_8d00b4c04db9.slice/crio-ecc2b14d09d63075cbe3d92465c10b6003d0677d390801b86a553240186fc8db WatchSource:0}: Error finding container ecc2b14d09d63075cbe3d92465c10b6003d0677d390801b86a553240186fc8db: Status 404 returned error can't find the container with id ecc2b14d09d63075cbe3d92465c10b6003d0677d390801b86a553240186fc8db
Jul 17 19:14:41.351053 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:41.351024    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw" event={"ID":"cbaaa271-a381-4d42-b517-8d00b4c04db9","Type":"ContainerStarted","Data":"ecc2b14d09d63075cbe3d92465c10b6003d0677d390801b86a553240186fc8db"}
Jul 17 19:14:42.359447 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:42.359409    2559 generic.go:356] "Generic (PLEG): container finished" podID="cbaaa271-a381-4d42-b517-8d00b4c04db9" containerID="2bafb0752c83dd699b87c803ff9bd545e1afa5842bb66e9a7062618a465e50c5" exitCode=7
Jul 17 19:14:42.359870 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:42.359495    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw" event={"ID":"cbaaa271-a381-4d42-b517-8d00b4c04db9","Type":"ContainerDied","Data":"2bafb0752c83dd699b87c803ff9bd545e1afa5842bb66e9a7062618a465e50c5"}
Jul 17 19:14:42.359870 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:42.359665    2559 scope.go:117] "RemoveContainer" containerID="2bafb0752c83dd699b87c803ff9bd545e1afa5842bb66e9a7062618a465e50c5"
Jul 17 19:14:43.366716 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:43.366680    2559 generic.go:356] "Generic (PLEG): container finished" podID="cbaaa271-a381-4d42-b517-8d00b4c04db9" containerID="0ada630c4dd542761ad3b7f9bb2a0ce366ba0acab5da5d93d86ca527c79b8bac" exitCode=7
Jul 17 19:14:43.367114 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:43.366744    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw" event={"ID":"cbaaa271-a381-4d42-b517-8d00b4c04db9","Type":"ContainerDied","Data":"0ada630c4dd542761ad3b7f9bb2a0ce366ba0acab5da5d93d86ca527c79b8bac"}
Jul 17 19:14:43.367114 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:43.366791    2559 scope.go:117] "RemoveContainer" containerID="2bafb0752c83dd699b87c803ff9bd545e1afa5842bb66e9a7062618a465e50c5"
Jul 17 19:14:43.367114 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:43.366921    2559 scope.go:117] "RemoveContainer" containerID="0ada630c4dd542761ad3b7f9bb2a0ce366ba0acab5da5d93d86ca527c79b8bac"
Jul 17 19:14:43.367295 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:14:43.367114    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw_odh-ai-gateway-infra(cbaaa271-a381-4d42-b517-8d00b4c04db9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw" podUID="cbaaa271-a381-4d42-b517-8d00b4c04db9"
Jul 17 19:14:44.373977 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:44.373953    2559 scope.go:117] "RemoveContainer" containerID="0ada630c4dd542761ad3b7f9bb2a0ce366ba0acab5da5d93d86ca527c79b8bac"
Jul 17 19:14:44.374552 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:14:44.374175    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw_odh-ai-gateway-infra(cbaaa271-a381-4d42-b517-8d00b4c04db9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw" podUID="cbaaa271-a381-4d42-b517-8d00b4c04db9"
Jul 17 19:14:51.120165 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:51.120133    2559 scope.go:117] "RemoveContainer" containerID="b4ff4031879e1fe3e3ef7371ffcf6e23a3b480f612eab7ebb3e1dd03d83b6f26"
Jul 17 19:14:51.416675 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:51.416607    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h" event={"ID":"0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492","Type":"ContainerStarted","Data":"8319f30424e854c47b6bee9f29980df9aafda11be20af6a138aa96e839c1e598"}
Jul 17 19:14:51.434147 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:51.434098    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h" podStartSLOduration=18.434083462 podStartE2EDuration="18.434083462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:14:51.432982473 +0000 UTC m=+2501.874491163" watchObservedRunningTime="2026-07-17 19:14:51.434083462 +0000 UTC m=+2501.875592150"
Jul 17 19:14:52.423755 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:52.423713    2559 generic.go:356] "Generic (PLEG): container finished" podID="0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492" containerID="8319f30424e854c47b6bee9f29980df9aafda11be20af6a138aa96e839c1e598" exitCode=6
Jul 17 19:14:52.424188 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:52.423783    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h" event={"ID":"0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492","Type":"ContainerDied","Data":"8319f30424e854c47b6bee9f29980df9aafda11be20af6a138aa96e839c1e598"}
Jul 17 19:14:52.424188 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:52.423843    2559 scope.go:117] "RemoveContainer" containerID="b4ff4031879e1fe3e3ef7371ffcf6e23a3b480f612eab7ebb3e1dd03d83b6f26"
Jul 17 19:14:52.424188 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:52.424125    2559 scope.go:117] "RemoveContainer" containerID="8319f30424e854c47b6bee9f29980df9aafda11be20af6a138aa96e839c1e598"
Jul 17 19:14:52.424386 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:14:52.424366    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h_odh-ai-gateway-infra(0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h" podUID="0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492"
Jul 17 19:14:52.446491 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:52.446465    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h"]
Jul 17 19:14:53.565832 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:53.565806    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h"
Jul 17 19:14:53.620038 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:53.620008    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492-maas-api-service-ca\") pod \"0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492\" (UID: \"0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492\") "
Jul 17 19:14:53.620387 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:53.620362    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492" (UID: "0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:14:53.721576 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:53.721495    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:14:54.438360 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:54.438256    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h" event={"ID":"0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492","Type":"ContainerDied","Data":"6ad663050a9906450e54b685d12bded3b2100034aafc0a6fd43a2d1b569fd610"}
Jul 17 19:14:54.438360 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:54.438341    2559 scope.go:117] "RemoveContainer" containerID="8319f30424e854c47b6bee9f29980df9aafda11be20af6a138aa96e839c1e598"
Jul 17 19:14:54.438360 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:54.438274    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h"
Jul 17 19:14:54.455415 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:54.455387    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h"]
Jul 17 19:14:54.460856 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:54.460826    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ksc2h"]
Jul 17 19:14:55.552144 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:55.552113    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk"]
Jul 17 19:14:55.552595 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:55.552487    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492" containerName="revoke-keys"
Jul 17 19:14:55.552595 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:55.552498    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492" containerName="revoke-keys"
Jul 17 19:14:55.552595 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:55.552507    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492" containerName="revoke-keys"
Jul 17 19:14:55.552595 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:55.552513    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492" containerName="revoke-keys"
Jul 17 19:14:55.552595 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:55.552587    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492" containerName="revoke-keys"
Jul 17 19:14:55.552595 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:55.552595    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492" containerName="revoke-keys"
Jul 17 19:14:55.552822 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:55.552601    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492" containerName="revoke-keys"
Jul 17 19:14:55.556818 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:55.556798    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk"
Jul 17 19:14:55.563229 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:55.563206    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk"]
Jul 17 19:14:55.637567 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:55.637543    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f5f871-7eb0-4644-a006-0106924e7dbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk\" (UID: \"78f5f871-7eb0-4644-a006-0106924e7dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk"
Jul 17 19:14:55.738743 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:55.738715    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f5f871-7eb0-4644-a006-0106924e7dbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk\" (UID: \"78f5f871-7eb0-4644-a006-0106924e7dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk"
Jul 17 19:14:55.739299 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:55.739282    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f5f871-7eb0-4644-a006-0106924e7dbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk\" (UID: \"78f5f871-7eb0-4644-a006-0106924e7dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk"
Jul 17 19:14:55.869102 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:55.869069    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk"
Jul 17 19:14:55.999580 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:55.999555    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk"]
Jul 17 19:14:56.001427 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:14:56.001394    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f5f871_7eb0_4644_a006_0106924e7dbd.slice/crio-c9aa69dbe830b749a67af61567e78cfdf8a3be5e6d7ce0b1d8c9cf68b169ca5f WatchSource:0}: Error finding container c9aa69dbe830b749a67af61567e78cfdf8a3be5e6d7ce0b1d8c9cf68b169ca5f: Status 404 returned error can't find the container with id c9aa69dbe830b749a67af61567e78cfdf8a3be5e6d7ce0b1d8c9cf68b169ca5f
Jul 17 19:14:56.125781 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:56.125710    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492" path="/var/lib/kubelet/pods/0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492/volumes"
Jul 17 19:14:56.451254 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:56.451179    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk" event={"ID":"78f5f871-7eb0-4644-a006-0106924e7dbd","Type":"ContainerStarted","Data":"c16b7acb257f423cb25acc03b5dd76e197ffe963e28d46b6199bcb0fb0f8a6ac"}
Jul 17 19:14:56.451254 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:56.451233    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk" event={"ID":"78f5f871-7eb0-4644-a006-0106924e7dbd","Type":"ContainerStarted","Data":"c9aa69dbe830b749a67af61567e78cfdf8a3be5e6d7ce0b1d8c9cf68b169ca5f"}
Jul 17 19:14:56.469490 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:56.469437    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk" podStartSLOduration=1.4694245129999999 podStartE2EDuration="1.469424513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:14:56.467458841 +0000 UTC m=+2506.908967531" watchObservedRunningTime="2026-07-17 19:14:56.469424513 +0000 UTC m=+2506.910933201"
Jul 17 19:14:57.459000 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:57.458962    2559 generic.go:356] "Generic (PLEG): container finished" podID="78f5f871-7eb0-4644-a006-0106924e7dbd" containerID="c16b7acb257f423cb25acc03b5dd76e197ffe963e28d46b6199bcb0fb0f8a6ac" exitCode=6
Jul 17 19:14:57.459393 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:57.459047    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk" event={"ID":"78f5f871-7eb0-4644-a006-0106924e7dbd","Type":"ContainerDied","Data":"c16b7acb257f423cb25acc03b5dd76e197ffe963e28d46b6199bcb0fb0f8a6ac"}
Jul 17 19:14:57.459393 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:57.459272    2559 scope.go:117] "RemoveContainer" containerID="c16b7acb257f423cb25acc03b5dd76e197ffe963e28d46b6199bcb0fb0f8a6ac"
Jul 17 19:14:58.466703 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:58.466671    2559 generic.go:356] "Generic (PLEG): container finished" podID="78f5f871-7eb0-4644-a006-0106924e7dbd" containerID="5d936e18678cdef58ae34e79acb9b5ad5ff9ac356c07227d31f1526073edfbd9" exitCode=6
Jul 17 19:14:58.467169 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:58.466755    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk" event={"ID":"78f5f871-7eb0-4644-a006-0106924e7dbd","Type":"ContainerDied","Data":"5d936e18678cdef58ae34e79acb9b5ad5ff9ac356c07227d31f1526073edfbd9"}
Jul 17 19:14:58.467169 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:58.466823    2559 scope.go:117] "RemoveContainer" containerID="c16b7acb257f423cb25acc03b5dd76e197ffe963e28d46b6199bcb0fb0f8a6ac"
Jul 17 19:14:58.467169 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:58.467028    2559 scope.go:117] "RemoveContainer" containerID="5d936e18678cdef58ae34e79acb9b5ad5ff9ac356c07227d31f1526073edfbd9"
Jul 17 19:14:58.467359 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:14:58.467275    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk_odh-ai-gateway-infra(78f5f871-7eb0-4644-a006-0106924e7dbd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk" podUID="78f5f871-7eb0-4644-a006-0106924e7dbd"
Jul 17 19:14:59.120277 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:59.120246    2559 scope.go:117] "RemoveContainer" containerID="0ada630c4dd542761ad3b7f9bb2a0ce366ba0acab5da5d93d86ca527c79b8bac"
Jul 17 19:14:59.473476 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:59.473452    2559 scope.go:117] "RemoveContainer" containerID="5d936e18678cdef58ae34e79acb9b5ad5ff9ac356c07227d31f1526073edfbd9"
Jul 17 19:14:59.473822 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:14:59.473627    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk_odh-ai-gateway-infra(78f5f871-7eb0-4644-a006-0106924e7dbd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk" podUID="78f5f871-7eb0-4644-a006-0106924e7dbd"
Jul 17 19:14:59.475137 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:59.475116    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw" event={"ID":"cbaaa271-a381-4d42-b517-8d00b4c04db9","Type":"ContainerStarted","Data":"c8b4a070487a00eac706731b25a0daa1dbc93925871df9de8cd2bd49aacf6ee5"}
Jul 17 19:14:59.505609 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:14:59.505570    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw" podStartSLOduration=19.505556825 podStartE2EDuration="19.505556825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:14:59.504655853 +0000 UTC m=+2509.946164545" watchObservedRunningTime="2026-07-17 19:14:59.505556825 +0000 UTC m=+2509.947065514"
Jul 17 19:15:00.136962 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:00.136881    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738595-hwqmv"]
Jul 17 19:15:00.137410 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:00.137393    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492" containerName="revoke-keys"
Jul 17 19:15:00.137410 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:00.137411    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fbc9dd6-7ae3-4a49-9aa0-edc5c4ee2492" containerName="revoke-keys"
Jul 17 19:15:00.140417 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:00.140397    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7"]
Jul 17 19:15:00.140539 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:00.140514    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738595-hwqmv"
Jul 17 19:15:00.143537 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:00.143521    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7"
Jul 17 19:15:00.150230 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:00.149955    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738595-hwqmv"]
Jul 17 19:15:00.153472 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:00.153446    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7"]
Jul 17 19:15:00.153949 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:00.153933    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738595-hwqmv"
Jul 17 19:15:00.161336 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:00.161293    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7"
Jul 17 19:15:00.307898 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:00.307870    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738595-hwqmv"]
Jul 17 19:15:00.309991 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:15:00.309940    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc4998a_504f_4a3f_af90_8d5f03d500a3.slice/crio-9bcc0b70b774097c8563bab2ac0bf44f5bb22e430594519e07d11ff22ac7efe0 WatchSource:0}: Error finding container 9bcc0b70b774097c8563bab2ac0bf44f5bb22e430594519e07d11ff22ac7efe0: Status 404 returned error can't find the container with id 9bcc0b70b774097c8563bab2ac0bf44f5bb22e430594519e07d11ff22ac7efe0
Jul 17 19:15:00.336028 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:00.336002    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7"]
Jul 17 19:15:00.337524 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:15:00.337492    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c172a3_50ff_4ee0_b1d7_847f9eb66afa.slice/crio-7df8c0ba78b651ebf235ba8f2a657160c9c6e389b0b5589b8fc2310351d6539e WatchSource:0}: Error finding container 7df8c0ba78b651ebf235ba8f2a657160c9c6e389b0b5589b8fc2310351d6539e: Status 404 returned error can't find the container with id 7df8c0ba78b651ebf235ba8f2a657160c9c6e389b0b5589b8fc2310351d6539e
Jul 17 19:15:00.483474 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:00.483389    2559 generic.go:356] "Generic (PLEG): container finished" podID="cbaaa271-a381-4d42-b517-8d00b4c04db9" containerID="c8b4a070487a00eac706731b25a0daa1dbc93925871df9de8cd2bd49aacf6ee5" exitCode=7
Jul 17 19:15:00.483474 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:00.483465    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw" event={"ID":"cbaaa271-a381-4d42-b517-8d00b4c04db9","Type":"ContainerDied","Data":"c8b4a070487a00eac706731b25a0daa1dbc93925871df9de8cd2bd49aacf6ee5"}
Jul 17 19:15:00.483986 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:00.483521    2559 scope.go:117] "RemoveContainer" containerID="0ada630c4dd542761ad3b7f9bb2a0ce366ba0acab5da5d93d86ca527c79b8bac"
Jul 17 19:15:00.483986 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:00.483829    2559 scope.go:117] "RemoveContainer" containerID="c8b4a070487a00eac706731b25a0daa1dbc93925871df9de8cd2bd49aacf6ee5"
Jul 17 19:15:00.484072 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:15:00.484038    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw_odh-ai-gateway-infra(cbaaa271-a381-4d42-b517-8d00b4c04db9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw" podUID="cbaaa271-a381-4d42-b517-8d00b4c04db9"
Jul 17 19:15:00.484897 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:00.484878    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738595-hwqmv" event={"ID":"dfc4998a-504f-4a3f-af90-8d5f03d500a3","Type":"ContainerStarted","Data":"9bcc0b70b774097c8563bab2ac0bf44f5bb22e430594519e07d11ff22ac7efe0"}
Jul 17 19:15:00.486138 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:00.486117    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7" event={"ID":"60c172a3-50ff-4ee0-b1d7-847f9eb66afa","Type":"ContainerStarted","Data":"7df8c0ba78b651ebf235ba8f2a657160c9c6e389b0b5589b8fc2310351d6539e"}
Jul 17 19:15:00.518061 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:00.518037    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw"]
Jul 17 19:15:01.492959 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:01.492927    2559 generic.go:356] "Generic (PLEG): container finished" podID="60c172a3-50ff-4ee0-b1d7-847f9eb66afa" containerID="edd88cef4a9881b4e8dd1a27963c4f80effb6a1df0f853ccc99201d3647adaf0" exitCode=7
Jul 17 19:15:01.493423 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:01.493020    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7" event={"ID":"60c172a3-50ff-4ee0-b1d7-847f9eb66afa","Type":"ContainerDied","Data":"edd88cef4a9881b4e8dd1a27963c4f80effb6a1df0f853ccc99201d3647adaf0"}
Jul 17 19:15:01.493423 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:01.493168    2559 scope.go:117] "RemoveContainer" containerID="edd88cef4a9881b4e8dd1a27963c4f80effb6a1df0f853ccc99201d3647adaf0"
Jul 17 19:15:01.496265 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:01.496234    2559 generic.go:356] "Generic (PLEG): container finished" podID="dfc4998a-504f-4a3f-af90-8d5f03d500a3" containerID="1195e4fea19bf32869a9a8ae265dce2e664d9d62a6612d015b257b36eb4183c5" exitCode=0
Jul 17 19:15:01.496389 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:01.496287    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738595-hwqmv" event={"ID":"dfc4998a-504f-4a3f-af90-8d5f03d500a3","Type":"ContainerDied","Data":"1195e4fea19bf32869a9a8ae265dce2e664d9d62a6612d015b257b36eb4183c5"}
Jul 17 19:15:01.634752 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:01.634727    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw"
Jul 17 19:15:01.691693 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:01.691666    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbaaa271-a381-4d42-b517-8d00b4c04db9-maas-api-service-ca\") pod \"cbaaa271-a381-4d42-b517-8d00b4c04db9\" (UID: \"cbaaa271-a381-4d42-b517-8d00b4c04db9\") "
Jul 17 19:15:01.692026 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:01.692003    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbaaa271-a381-4d42-b517-8d00b4c04db9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cbaaa271-a381-4d42-b517-8d00b4c04db9" (UID: "cbaaa271-a381-4d42-b517-8d00b4c04db9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:15:01.792801 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:01.792772    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbaaa271-a381-4d42-b517-8d00b4c04db9-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:15:02.503632 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.503539    2559 generic.go:356] "Generic (PLEG): container finished" podID="60c172a3-50ff-4ee0-b1d7-847f9eb66afa" containerID="e9066add572c3f96564caf2871754e172d985f20be6e9903dcd43c54bad3f146" exitCode=7
Jul 17 19:15:02.503632 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.503596    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7" event={"ID":"60c172a3-50ff-4ee0-b1d7-847f9eb66afa","Type":"ContainerDied","Data":"e9066add572c3f96564caf2871754e172d985f20be6e9903dcd43c54bad3f146"}
Jul 17 19:15:02.504128 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.503652    2559 scope.go:117] "RemoveContainer" containerID="edd88cef4a9881b4e8dd1a27963c4f80effb6a1df0f853ccc99201d3647adaf0"
Jul 17 19:15:02.504128 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.503794    2559 scope.go:117] "RemoveContainer" containerID="e9066add572c3f96564caf2871754e172d985f20be6e9903dcd43c54bad3f146"
Jul 17 19:15:02.504128 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:15:02.504014    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7_odh-ai-gateway-infra(60c172a3-50ff-4ee0-b1d7-847f9eb66afa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7" podUID="60c172a3-50ff-4ee0-b1d7-847f9eb66afa"
Jul 17 19:15:02.505014 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.504968    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw"
Jul 17 19:15:02.505085 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.505017    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw" event={"ID":"cbaaa271-a381-4d42-b517-8d00b4c04db9","Type":"ContainerDied","Data":"ecc2b14d09d63075cbe3d92465c10b6003d0677d390801b86a553240186fc8db"}
Jul 17 19:15:02.517189 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.517136    2559 scope.go:117] "RemoveContainer" containerID="c8b4a070487a00eac706731b25a0daa1dbc93925871df9de8cd2bd49aacf6ee5"
Jul 17 19:15:02.531411 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.531383    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw"]
Jul 17 19:15:02.535368 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.535343    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-l2wbw"]
Jul 17 19:15:02.567457 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.567423    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh"]
Jul 17 19:15:02.568018 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.567996    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbaaa271-a381-4d42-b517-8d00b4c04db9" containerName="revoke-keys"
Jul 17 19:15:02.568018 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.568018    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbaaa271-a381-4d42-b517-8d00b4c04db9" containerName="revoke-keys"
Jul 17 19:15:02.568180 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.568047    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbaaa271-a381-4d42-b517-8d00b4c04db9" containerName="revoke-keys"
Jul 17 19:15:02.568180 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.568055    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbaaa271-a381-4d42-b517-8d00b4c04db9" containerName="revoke-keys"
Jul 17 19:15:02.568180 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.568145    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbaaa271-a381-4d42-b517-8d00b4c04db9" containerName="revoke-keys"
Jul 17 19:15:02.568180 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.568158    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbaaa271-a381-4d42-b517-8d00b4c04db9" containerName="revoke-keys"
Jul 17 19:15:02.572823 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.572803    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh"
Jul 17 19:15:02.580843 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.580757    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh"]
Jul 17 19:15:02.646253 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.646231    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738595-hwqmv"
Jul 17 19:15:02.701512 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.701480    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab68b4f1-3a12-4c5c-a481-559fffc99b9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh\" (UID: \"ab68b4f1-3a12-4c5c-a481-559fffc99b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh"
Jul 17 19:15:02.802326 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.802215    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab68b4f1-3a12-4c5c-a481-559fffc99b9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh\" (UID: \"ab68b4f1-3a12-4c5c-a481-559fffc99b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh"
Jul 17 19:15:02.802838 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.802817    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab68b4f1-3a12-4c5c-a481-559fffc99b9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh\" (UID: \"ab68b4f1-3a12-4c5c-a481-559fffc99b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh"
Jul 17 19:15:02.886807 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:02.886777    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh"
Jul 17 19:15:03.020640 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:03.020614    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh"]
Jul 17 19:15:03.022573 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:15:03.022545    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab68b4f1_3a12_4c5c_a481_559fffc99b9a.slice/crio-1fbfb5adcd511bfae669f235d51c8adf13e6540457a6883bfe2c52662f9dedf3 WatchSource:0}: Error finding container 1fbfb5adcd511bfae669f235d51c8adf13e6540457a6883bfe2c52662f9dedf3: Status 404 returned error can't find the container with id 1fbfb5adcd511bfae669f235d51c8adf13e6540457a6883bfe2c52662f9dedf3
Jul 17 19:15:03.513392 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:03.513366    2559 scope.go:117] "RemoveContainer" containerID="e9066add572c3f96564caf2871754e172d985f20be6e9903dcd43c54bad3f146"
Jul 17 19:15:03.513826 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:15:03.513586    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7_odh-ai-gateway-infra(60c172a3-50ff-4ee0-b1d7-847f9eb66afa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7" podUID="60c172a3-50ff-4ee0-b1d7-847f9eb66afa"
Jul 17 19:15:03.515269 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:03.515250    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738595-hwqmv"
Jul 17 19:15:03.515400 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:03.515278    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738595-hwqmv" event={"ID":"dfc4998a-504f-4a3f-af90-8d5f03d500a3","Type":"ContainerDied","Data":"9bcc0b70b774097c8563bab2ac0bf44f5bb22e430594519e07d11ff22ac7efe0"}
Jul 17 19:15:03.515400 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:03.515342    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bcc0b70b774097c8563bab2ac0bf44f5bb22e430594519e07d11ff22ac7efe0"
Jul 17 19:15:03.516780 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:03.516755    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh" event={"ID":"ab68b4f1-3a12-4c5c-a481-559fffc99b9a","Type":"ContainerStarted","Data":"d1513c107cbac754ec07d584dc0a2c3465b776dae13ef68bc096fa2b3a9b1263"}
Jul 17 19:15:03.516906 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:03.516793    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh" event={"ID":"ab68b4f1-3a12-4c5c-a481-559fffc99b9a","Type":"ContainerStarted","Data":"1fbfb5adcd511bfae669f235d51c8adf13e6540457a6883bfe2c52662f9dedf3"}
Jul 17 19:15:03.547453 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:03.547413    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh" podStartSLOduration=1.547400208 podStartE2EDuration="1.547400208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:15:03.544414486 +0000 UTC m=+2513.985923171" watchObservedRunningTime="2026-07-17 19:15:03.547400208 +0000 UTC m=+2513.988908898"
Jul 17 19:15:04.125567 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:04.125535    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbaaa271-a381-4d42-b517-8d00b4c04db9" path="/var/lib/kubelet/pods/cbaaa271-a381-4d42-b517-8d00b4c04db9/volumes"
Jul 17 19:15:04.530047 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:04.529961    2559 generic.go:356] "Generic (PLEG): container finished" podID="ab68b4f1-3a12-4c5c-a481-559fffc99b9a" containerID="d1513c107cbac754ec07d584dc0a2c3465b776dae13ef68bc096fa2b3a9b1263" exitCode=6
Jul 17 19:15:04.530458 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:04.530053    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh" event={"ID":"ab68b4f1-3a12-4c5c-a481-559fffc99b9a","Type":"ContainerDied","Data":"d1513c107cbac754ec07d584dc0a2c3465b776dae13ef68bc096fa2b3a9b1263"}
Jul 17 19:15:04.530458 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:04.530181    2559 scope.go:117] "RemoveContainer" containerID="d1513c107cbac754ec07d584dc0a2c3465b776dae13ef68bc096fa2b3a9b1263"
Jul 17 19:15:05.538383 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:05.538287    2559 generic.go:356] "Generic (PLEG): container finished" podID="ab68b4f1-3a12-4c5c-a481-559fffc99b9a" containerID="3f23f49c66763597d28ed44f1cde20543f0bcd4402e6fae30b7c28716fee8759" exitCode=6
Jul 17 19:15:05.538383 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:05.538351    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh" event={"ID":"ab68b4f1-3a12-4c5c-a481-559fffc99b9a","Type":"ContainerDied","Data":"3f23f49c66763597d28ed44f1cde20543f0bcd4402e6fae30b7c28716fee8759"}
Jul 17 19:15:05.538818 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:05.538427    2559 scope.go:117] "RemoveContainer" containerID="d1513c107cbac754ec07d584dc0a2c3465b776dae13ef68bc096fa2b3a9b1263"
Jul 17 19:15:05.538818 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:05.538633    2559 scope.go:117] "RemoveContainer" containerID="3f23f49c66763597d28ed44f1cde20543f0bcd4402e6fae30b7c28716fee8759"
Jul 17 19:15:05.538907 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:15:05.538875    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh_odh-ai-gateway-infra(ab68b4f1-3a12-4c5c-a481-559fffc99b9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh" podUID="ab68b4f1-3a12-4c5c-a481-559fffc99b9a"
Jul 17 19:15:06.547116 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:06.547090    2559 scope.go:117] "RemoveContainer" containerID="3f23f49c66763597d28ed44f1cde20543f0bcd4402e6fae30b7c28716fee8759"
Jul 17 19:15:06.547501 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:15:06.547272    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh_odh-ai-gateway-infra(ab68b4f1-3a12-4c5c-a481-559fffc99b9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh" podUID="ab68b4f1-3a12-4c5c-a481-559fffc99b9a"
Jul 17 19:15:13.119365 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:13.119334    2559 scope.go:117] "RemoveContainer" containerID="5d936e18678cdef58ae34e79acb9b5ad5ff9ac356c07227d31f1526073edfbd9"
Jul 17 19:15:13.587362 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:13.587337    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk" event={"ID":"78f5f871-7eb0-4644-a006-0106924e7dbd","Type":"ContainerStarted","Data":"381ece127ae8a213aa41843afb4ed700d8d3da04ee8cc915ab3b3abe29b7be34"}
Jul 17 19:15:14.126194 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:14.126171    2559 scope.go:117] "RemoveContainer" containerID="e9066add572c3f96564caf2871754e172d985f20be6e9903dcd43c54bad3f146"
Jul 17 19:15:14.594730 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:14.594696    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7" event={"ID":"60c172a3-50ff-4ee0-b1d7-847f9eb66afa","Type":"ContainerStarted","Data":"74fd91256f87d4d3cbe27046a6b373cd33ce848c0691f6d5e853a4086fc9823b"}
Jul 17 19:15:14.596384 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:14.596357    2559 generic.go:356] "Generic (PLEG): container finished" podID="78f5f871-7eb0-4644-a006-0106924e7dbd" containerID="381ece127ae8a213aa41843afb4ed700d8d3da04ee8cc915ab3b3abe29b7be34" exitCode=6
Jul 17 19:15:14.596496 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:14.596398    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk" event={"ID":"78f5f871-7eb0-4644-a006-0106924e7dbd","Type":"ContainerDied","Data":"381ece127ae8a213aa41843afb4ed700d8d3da04ee8cc915ab3b3abe29b7be34"}
Jul 17 19:15:14.596496 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:14.596445    2559 scope.go:117] "RemoveContainer" containerID="5d936e18678cdef58ae34e79acb9b5ad5ff9ac356c07227d31f1526073edfbd9"
Jul 17 19:15:14.596770 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:14.596752    2559 scope.go:117] "RemoveContainer" containerID="381ece127ae8a213aa41843afb4ed700d8d3da04ee8cc915ab3b3abe29b7be34"
Jul 17 19:15:14.596990 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:15:14.596971    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk_odh-ai-gateway-infra(78f5f871-7eb0-4644-a006-0106924e7dbd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk" podUID="78f5f871-7eb0-4644-a006-0106924e7dbd"
Jul 17 19:15:14.611447 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:14.611404    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7" podStartSLOduration=14.611390504 podStartE2EDuration="14.611390504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:15:14.609521282 +0000 UTC m=+2525.051029976" watchObservedRunningTime="2026-07-17 19:15:14.611390504 +0000 UTC m=+2525.052899193"
Jul 17 19:15:14.615839 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:14.615817    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk"]
Jul 17 19:15:15.604624 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:15.604586    2559 generic.go:356] "Generic (PLEG): container finished" podID="60c172a3-50ff-4ee0-b1d7-847f9eb66afa" containerID="74fd91256f87d4d3cbe27046a6b373cd33ce848c0691f6d5e853a4086fc9823b" exitCode=7
Jul 17 19:15:15.605225 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:15.604661    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7" event={"ID":"60c172a3-50ff-4ee0-b1d7-847f9eb66afa","Type":"ContainerDied","Data":"74fd91256f87d4d3cbe27046a6b373cd33ce848c0691f6d5e853a4086fc9823b"}
Jul 17 19:15:15.605225 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:15.604725    2559 scope.go:117] "RemoveContainer" containerID="e9066add572c3f96564caf2871754e172d985f20be6e9903dcd43c54bad3f146"
Jul 17 19:15:15.605225 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:15.604968    2559 scope.go:117] "RemoveContainer" containerID="74fd91256f87d4d3cbe27046a6b373cd33ce848c0691f6d5e853a4086fc9823b"
Jul 17 19:15:15.605447 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:15:15.605262    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7_odh-ai-gateway-infra(60c172a3-50ff-4ee0-b1d7-847f9eb66afa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7" podUID="60c172a3-50ff-4ee0-b1d7-847f9eb66afa"
Jul 17 19:15:15.622286 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:15.622258    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7"]
Jul 17 19:15:15.747289 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:15.747265    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk"
Jul 17 19:15:15.825650 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:15.825619    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f5f871-7eb0-4644-a006-0106924e7dbd-maas-api-service-ca\") pod \"78f5f871-7eb0-4644-a006-0106924e7dbd\" (UID: \"78f5f871-7eb0-4644-a006-0106924e7dbd\") "
Jul 17 19:15:15.825945 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:15.825922    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78f5f871-7eb0-4644-a006-0106924e7dbd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78f5f871-7eb0-4644-a006-0106924e7dbd" (UID: "78f5f871-7eb0-4644-a006-0106924e7dbd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:15:15.926966 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:15.926883    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f5f871-7eb0-4644-a006-0106924e7dbd-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:15:16.613772 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:16.613745    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk"
Jul 17 19:15:16.614252 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:16.613787    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk" event={"ID":"78f5f871-7eb0-4644-a006-0106924e7dbd","Type":"ContainerDied","Data":"c9aa69dbe830b749a67af61567e78cfdf8a3be5e6d7ce0b1d8c9cf68b169ca5f"}
Jul 17 19:15:16.614252 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:16.613850    2559 scope.go:117] "RemoveContainer" containerID="381ece127ae8a213aa41843afb4ed700d8d3da04ee8cc915ab3b3abe29b7be34"
Jul 17 19:15:16.631357 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:16.631331    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk"]
Jul 17 19:15:16.634525 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:16.634497    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-blrbk"]
Jul 17 19:15:16.753836 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:16.753814    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7"
Jul 17 19:15:17.622533 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:17.622505    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7"
Jul 17 19:15:17.622533 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:17.622517    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7" event={"ID":"60c172a3-50ff-4ee0-b1d7-847f9eb66afa","Type":"ContainerDied","Data":"7df8c0ba78b651ebf235ba8f2a657160c9c6e389b0b5589b8fc2310351d6539e"}
Jul 17 19:15:17.623083 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:17.622584    2559 scope.go:117] "RemoveContainer" containerID="74fd91256f87d4d3cbe27046a6b373cd33ce848c0691f6d5e853a4086fc9823b"
Jul 17 19:15:17.646101 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:17.646076    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7"]
Jul 17 19:15:17.650757 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:17.650728    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-b74adac0-29738595-pptk7"]
Jul 17 19:15:18.125281 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:18.125246    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60c172a3-50ff-4ee0-b1d7-847f9eb66afa" path="/var/lib/kubelet/pods/60c172a3-50ff-4ee0-b1d7-847f9eb66afa/volumes"
Jul 17 19:15:18.125595 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:18.125580    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78f5f871-7eb0-4644-a006-0106924e7dbd" path="/var/lib/kubelet/pods/78f5f871-7eb0-4644-a006-0106924e7dbd/volumes"
Jul 17 19:15:21.119868 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:21.119841    2559 scope.go:117] "RemoveContainer" containerID="3f23f49c66763597d28ed44f1cde20543f0bcd4402e6fae30b7c28716fee8759"
Jul 17 19:15:21.655953 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:21.655920    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh" event={"ID":"ab68b4f1-3a12-4c5c-a481-559fffc99b9a","Type":"ContainerStarted","Data":"72abab688507dede934b08de4b8a5ee8cbe0ccd90690401dce7b64b2fe4ec1ba"}
Jul 17 19:15:22.662863 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:22.662832    2559 generic.go:356] "Generic (PLEG): container finished" podID="ab68b4f1-3a12-4c5c-a481-559fffc99b9a" containerID="72abab688507dede934b08de4b8a5ee8cbe0ccd90690401dce7b64b2fe4ec1ba" exitCode=6
Jul 17 19:15:22.663342 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:22.662916    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh" event={"ID":"ab68b4f1-3a12-4c5c-a481-559fffc99b9a","Type":"ContainerDied","Data":"72abab688507dede934b08de4b8a5ee8cbe0ccd90690401dce7b64b2fe4ec1ba"}
Jul 17 19:15:22.663342 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:22.662980    2559 scope.go:117] "RemoveContainer" containerID="3f23f49c66763597d28ed44f1cde20543f0bcd4402e6fae30b7c28716fee8759"
Jul 17 19:15:22.663342 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:22.663267    2559 scope.go:117] "RemoveContainer" containerID="72abab688507dede934b08de4b8a5ee8cbe0ccd90690401dce7b64b2fe4ec1ba"
Jul 17 19:15:22.663519 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:15:22.663496    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh_odh-ai-gateway-infra(ab68b4f1-3a12-4c5c-a481-559fffc99b9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh" podUID="ab68b4f1-3a12-4c5c-a481-559fffc99b9a"
Jul 17 19:15:22.687471 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:22.687447    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh"]
Jul 17 19:15:23.805281 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:23.805251    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh"
Jul 17 19:15:23.999716 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:23.999627    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab68b4f1-3a12-4c5c-a481-559fffc99b9a-maas-api-service-ca\") pod \"ab68b4f1-3a12-4c5c-a481-559fffc99b9a\" (UID: \"ab68b4f1-3a12-4c5c-a481-559fffc99b9a\") "
Jul 17 19:15:23.999998 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:23.999972    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab68b4f1-3a12-4c5c-a481-559fffc99b9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ab68b4f1-3a12-4c5c-a481-559fffc99b9a" (UID: "ab68b4f1-3a12-4c5c-a481-559fffc99b9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:15:24.101100 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:24.101055    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab68b4f1-3a12-4c5c-a481-559fffc99b9a-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:15:24.678257 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:24.678226    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh"
Jul 17 19:15:24.680512 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:24.678262    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh" event={"ID":"ab68b4f1-3a12-4c5c-a481-559fffc99b9a","Type":"ContainerDied","Data":"1fbfb5adcd511bfae669f235d51c8adf13e6540457a6883bfe2c52662f9dedf3"}
Jul 17 19:15:24.680512 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:24.678366    2559 scope.go:117] "RemoveContainer" containerID="72abab688507dede934b08de4b8a5ee8cbe0ccd90690401dce7b64b2fe4ec1ba"
Jul 17 19:15:24.697522 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:24.697492    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh"]
Jul 17 19:15:24.699707 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:24.699684    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-lklfh"]
Jul 17 19:15:26.126201 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:26.126170    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab68b4f1-3a12-4c5c-a481-559fffc99b9a" path="/var/lib/kubelet/pods/ab68b4f1-3a12-4c5c-a481-559fffc99b9a/volumes"
Jul 17 19:15:33.732812 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.732782    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t"]
Jul 17 19:15:33.733236 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733186    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c172a3-50ff-4ee0-b1d7-847f9eb66afa" containerName="cleanup"
Jul 17 19:15:33.733236 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733198    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c172a3-50ff-4ee0-b1d7-847f9eb66afa" containerName="cleanup"
Jul 17 19:15:33.733236 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733207    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab68b4f1-3a12-4c5c-a481-559fffc99b9a" containerName="revoke-keys"
Jul 17 19:15:33.733236 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733213    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab68b4f1-3a12-4c5c-a481-559fffc99b9a" containerName="revoke-keys"
Jul 17 19:15:33.733236 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733225    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbaaa271-a381-4d42-b517-8d00b4c04db9" containerName="revoke-keys"
Jul 17 19:15:33.733236 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733230    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbaaa271-a381-4d42-b517-8d00b4c04db9" containerName="revoke-keys"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733241    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfc4998a-504f-4a3f-af90-8d5f03d500a3" containerName="cleanup"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733249    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfc4998a-504f-4a3f-af90-8d5f03d500a3" containerName="cleanup"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733263    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78f5f871-7eb0-4644-a006-0106924e7dbd" containerName="revoke-keys"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733268    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="78f5f871-7eb0-4644-a006-0106924e7dbd" containerName="revoke-keys"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733278    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78f5f871-7eb0-4644-a006-0106924e7dbd" containerName="revoke-keys"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733282    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="78f5f871-7eb0-4644-a006-0106924e7dbd" containerName="revoke-keys"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733295    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab68b4f1-3a12-4c5c-a481-559fffc99b9a" containerName="revoke-keys"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733299    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab68b4f1-3a12-4c5c-a481-559fffc99b9a" containerName="revoke-keys"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733317    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78f5f871-7eb0-4644-a006-0106924e7dbd" containerName="revoke-keys"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733322    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="78f5f871-7eb0-4644-a006-0106924e7dbd" containerName="revoke-keys"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733329    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c172a3-50ff-4ee0-b1d7-847f9eb66afa" containerName="cleanup"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733334    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c172a3-50ff-4ee0-b1d7-847f9eb66afa" containerName="cleanup"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733390    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab68b4f1-3a12-4c5c-a481-559fffc99b9a" containerName="revoke-keys"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733397    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c172a3-50ff-4ee0-b1d7-847f9eb66afa" containerName="cleanup"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733404    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="78f5f871-7eb0-4644-a006-0106924e7dbd" containerName="revoke-keys"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733414    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfc4998a-504f-4a3f-af90-8d5f03d500a3" containerName="cleanup"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733423    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbaaa271-a381-4d42-b517-8d00b4c04db9" containerName="revoke-keys"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733432    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab68b4f1-3a12-4c5c-a481-559fffc99b9a" containerName="revoke-keys"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733438    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="78f5f871-7eb0-4644-a006-0106924e7dbd" containerName="revoke-keys"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733443    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="78f5f871-7eb0-4644-a006-0106924e7dbd" containerName="revoke-keys"
Jul 17 19:15:33.733691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.733450    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c172a3-50ff-4ee0-b1d7-847f9eb66afa" containerName="cleanup"
Jul 17 19:15:33.736744 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.736728    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t"
Jul 17 19:15:33.739264 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.739246    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:15:33.743918 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.743892    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t"]
Jul 17 19:15:33.777759 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.777726    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fcc3021-0465-4e9a-af54-0de0638d778e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t\" (UID: \"6fcc3021-0465-4e9a-af54-0de0638d778e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t"
Jul 17 19:15:33.878845 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.878807    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fcc3021-0465-4e9a-af54-0de0638d778e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t\" (UID: \"6fcc3021-0465-4e9a-af54-0de0638d778e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t"
Jul 17 19:15:33.879445 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:33.879425    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fcc3021-0465-4e9a-af54-0de0638d778e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t\" (UID: \"6fcc3021-0465-4e9a-af54-0de0638d778e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t"
Jul 17 19:15:34.048433 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:34.048328    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t"
Jul 17 19:15:34.213608 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:34.213585    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t"]
Jul 17 19:15:34.215229 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:15:34.215203    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fcc3021_0465_4e9a_af54_0de0638d778e.slice/crio-d4af9567894cac623f2c6493111eb70dc633638dcd2f03b5b1086bb31584ba09 WatchSource:0}: Error finding container d4af9567894cac623f2c6493111eb70dc633638dcd2f03b5b1086bb31584ba09: Status 404 returned error can't find the container with id d4af9567894cac623f2c6493111eb70dc633638dcd2f03b5b1086bb31584ba09
Jul 17 19:15:34.737684 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:34.737598    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t" event={"ID":"6fcc3021-0465-4e9a-af54-0de0638d778e","Type":"ContainerStarted","Data":"2f28bc432e35471c99a657e0656b75d720504ef753d33dd3809c1680c73e5f89"}
Jul 17 19:15:34.737684 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:34.737658    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t" event={"ID":"6fcc3021-0465-4e9a-af54-0de0638d778e","Type":"ContainerStarted","Data":"d4af9567894cac623f2c6493111eb70dc633638dcd2f03b5b1086bb31584ba09"}
Jul 17 19:15:34.755951 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:34.755907    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t" podStartSLOduration=1.755893192 podStartE2EDuration="1.755893192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:15:34.754344921 +0000 UTC m=+2545.195853610" watchObservedRunningTime="2026-07-17 19:15:34.755893192 +0000 UTC m=+2545.197401880"
Jul 17 19:15:35.743812 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:35.743776    2559 generic.go:356] "Generic (PLEG): container finished" podID="6fcc3021-0465-4e9a-af54-0de0638d778e" containerID="2f28bc432e35471c99a657e0656b75d720504ef753d33dd3809c1680c73e5f89" exitCode=6
Jul 17 19:15:35.744184 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:35.743858    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t" event={"ID":"6fcc3021-0465-4e9a-af54-0de0638d778e","Type":"ContainerDied","Data":"2f28bc432e35471c99a657e0656b75d720504ef753d33dd3809c1680c73e5f89"}
Jul 17 19:15:35.744184 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:35.744009    2559 scope.go:117] "RemoveContainer" containerID="2f28bc432e35471c99a657e0656b75d720504ef753d33dd3809c1680c73e5f89"
Jul 17 19:15:36.751874 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:36.751774    2559 generic.go:356] "Generic (PLEG): container finished" podID="6fcc3021-0465-4e9a-af54-0de0638d778e" containerID="6aa1a63a26fc20ebc2d8cea36b3b0aada79f2d754133df3a1d125babcee19653" exitCode=6
Jul 17 19:15:36.751874 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:36.751812    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t" event={"ID":"6fcc3021-0465-4e9a-af54-0de0638d778e","Type":"ContainerDied","Data":"6aa1a63a26fc20ebc2d8cea36b3b0aada79f2d754133df3a1d125babcee19653"}
Jul 17 19:15:36.751874 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:36.751858    2559 scope.go:117] "RemoveContainer" containerID="2f28bc432e35471c99a657e0656b75d720504ef753d33dd3809c1680c73e5f89"
Jul 17 19:15:36.752508 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:36.752108    2559 scope.go:117] "RemoveContainer" containerID="6aa1a63a26fc20ebc2d8cea36b3b0aada79f2d754133df3a1d125babcee19653"
Jul 17 19:15:36.752508 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:15:36.752382    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t_odh-ai-gateway-infra(6fcc3021-0465-4e9a-af54-0de0638d778e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t" podUID="6fcc3021-0465-4e9a-af54-0de0638d778e"
Jul 17 19:15:37.759015 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:37.758983    2559 scope.go:117] "RemoveContainer" containerID="6aa1a63a26fc20ebc2d8cea36b3b0aada79f2d754133df3a1d125babcee19653"
Jul 17 19:15:37.759457 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:15:37.759205    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t_odh-ai-gateway-infra(6fcc3021-0465-4e9a-af54-0de0638d778e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t" podUID="6fcc3021-0465-4e9a-af54-0de0638d778e"
Jul 17 19:15:40.887856 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:40.887818    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9"]
Jul 17 19:15:40.888257 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:40.888244    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c172a3-50ff-4ee0-b1d7-847f9eb66afa" containerName="cleanup"
Jul 17 19:15:40.888257 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:40.888258    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c172a3-50ff-4ee0-b1d7-847f9eb66afa" containerName="cleanup"
Jul 17 19:15:40.888368 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:40.888266    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab68b4f1-3a12-4c5c-a481-559fffc99b9a" containerName="revoke-keys"
Jul 17 19:15:40.888368 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:40.888272    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab68b4f1-3a12-4c5c-a481-559fffc99b9a" containerName="revoke-keys"
Jul 17 19:15:40.888368 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:40.888345    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c172a3-50ff-4ee0-b1d7-847f9eb66afa" containerName="cleanup"
Jul 17 19:15:40.888368 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:40.888356    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab68b4f1-3a12-4c5c-a481-559fffc99b9a" containerName="revoke-keys"
Jul 17 19:15:40.892480 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:40.892465    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9"
Jul 17 19:15:40.900709 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:40.900438    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9"]
Jul 17 19:15:40.946697 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:40.946670    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7880eb14-169a-45fe-add7-0daaf9ad20b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-968j9\" (UID: \"7880eb14-169a-45fe-add7-0daaf9ad20b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9"
Jul 17 19:15:41.048041 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:41.047998    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7880eb14-169a-45fe-add7-0daaf9ad20b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-968j9\" (UID: \"7880eb14-169a-45fe-add7-0daaf9ad20b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9"
Jul 17 19:15:41.048651 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:41.048630    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7880eb14-169a-45fe-add7-0daaf9ad20b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-968j9\" (UID: \"7880eb14-169a-45fe-add7-0daaf9ad20b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9"
Jul 17 19:15:41.205732 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:41.205648    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9"
Jul 17 19:15:41.344406 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:41.344368    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9"]
Jul 17 19:15:41.346837 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:15:41.346809    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7880eb14_169a_45fe_add7_0daaf9ad20b5.slice/crio-b2d5cd0834ebd6ecc834b43a6cbec927284bbf567f946d145269a28f6194130a WatchSource:0}: Error finding container b2d5cd0834ebd6ecc834b43a6cbec927284bbf567f946d145269a28f6194130a: Status 404 returned error can't find the container with id b2d5cd0834ebd6ecc834b43a6cbec927284bbf567f946d145269a28f6194130a
Jul 17 19:15:41.782441 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:41.782403    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9" event={"ID":"7880eb14-169a-45fe-add7-0daaf9ad20b5","Type":"ContainerStarted","Data":"acf04a62924d44cc86d215aee417f7f6849299b4885521b3dca956b98646c921"}
Jul 17 19:15:41.782605 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:41.782463    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9" event={"ID":"7880eb14-169a-45fe-add7-0daaf9ad20b5","Type":"ContainerStarted","Data":"b2d5cd0834ebd6ecc834b43a6cbec927284bbf567f946d145269a28f6194130a"}
Jul 17 19:15:41.797115 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:41.797054    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9" podStartSLOduration=1.797039837 podStartE2EDuration="1.797039837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:15:41.796131843 +0000 UTC m=+2552.237640533" watchObservedRunningTime="2026-07-17 19:15:41.797039837 +0000 UTC m=+2552.238548562"
Jul 17 19:15:42.789629 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:42.789591    2559 generic.go:356] "Generic (PLEG): container finished" podID="7880eb14-169a-45fe-add7-0daaf9ad20b5" containerID="acf04a62924d44cc86d215aee417f7f6849299b4885521b3dca956b98646c921" exitCode=7
Jul 17 19:15:42.790016 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:42.789678    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9" event={"ID":"7880eb14-169a-45fe-add7-0daaf9ad20b5","Type":"ContainerDied","Data":"acf04a62924d44cc86d215aee417f7f6849299b4885521b3dca956b98646c921"}
Jul 17 19:15:42.790016 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:42.789866    2559 scope.go:117] "RemoveContainer" containerID="acf04a62924d44cc86d215aee417f7f6849299b4885521b3dca956b98646c921"
Jul 17 19:15:43.800797 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:43.800707    2559 generic.go:356] "Generic (PLEG): container finished" podID="7880eb14-169a-45fe-add7-0daaf9ad20b5" containerID="996f3408a8d3511afbddb622f168aa1fdc385d4c53d57093e0aa4d8aa2b5e3ae" exitCode=7
Jul 17 19:15:43.800797 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:43.800755    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9" event={"ID":"7880eb14-169a-45fe-add7-0daaf9ad20b5","Type":"ContainerDied","Data":"996f3408a8d3511afbddb622f168aa1fdc385d4c53d57093e0aa4d8aa2b5e3ae"}
Jul 17 19:15:43.801220 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:43.800839    2559 scope.go:117] "RemoveContainer" containerID="acf04a62924d44cc86d215aee417f7f6849299b4885521b3dca956b98646c921"
Jul 17 19:15:43.801220 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:43.801001    2559 scope.go:117] "RemoveContainer" containerID="996f3408a8d3511afbddb622f168aa1fdc385d4c53d57093e0aa4d8aa2b5e3ae"
Jul 17 19:15:43.801294 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:15:43.801272    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-968j9_odh-ai-gateway-infra(7880eb14-169a-45fe-add7-0daaf9ad20b5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9" podUID="7880eb14-169a-45fe-add7-0daaf9ad20b5"
Jul 17 19:15:44.809389 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:44.809357    2559 scope.go:117] "RemoveContainer" containerID="996f3408a8d3511afbddb622f168aa1fdc385d4c53d57093e0aa4d8aa2b5e3ae"
Jul 17 19:15:44.809750 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:15:44.809531    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-968j9_odh-ai-gateway-infra(7880eb14-169a-45fe-add7-0daaf9ad20b5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9" podUID="7880eb14-169a-45fe-add7-0daaf9ad20b5"
Jul 17 19:15:49.119898 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:49.119867    2559 scope.go:117] "RemoveContainer" containerID="6aa1a63a26fc20ebc2d8cea36b3b0aada79f2d754133df3a1d125babcee19653"
Jul 17 19:15:49.838345 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:49.838287    2559 generic.go:356] "Generic (PLEG): container finished" podID="6fcc3021-0465-4e9a-af54-0de0638d778e" containerID="b6ac2cbcb5602d6b00573d2b2f7b6841a71fe059ebd28ba3bcf97bf67edbfefb" exitCode=6
Jul 17 19:15:49.838515 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:49.838363    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t" event={"ID":"6fcc3021-0465-4e9a-af54-0de0638d778e","Type":"ContainerDied","Data":"b6ac2cbcb5602d6b00573d2b2f7b6841a71fe059ebd28ba3bcf97bf67edbfefb"}
Jul 17 19:15:49.838515 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:49.838456    2559 scope.go:117] "RemoveContainer" containerID="6aa1a63a26fc20ebc2d8cea36b3b0aada79f2d754133df3a1d125babcee19653"
Jul 17 19:15:49.838794 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:49.838771    2559 scope.go:117] "RemoveContainer" containerID="b6ac2cbcb5602d6b00573d2b2f7b6841a71fe059ebd28ba3bcf97bf67edbfefb"
Jul 17 19:15:49.839072 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:15:49.839033    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t_odh-ai-gateway-infra(6fcc3021-0465-4e9a-af54-0de0638d778e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t" podUID="6fcc3021-0465-4e9a-af54-0de0638d778e"
Jul 17 19:15:50.866120 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:50.866091    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t"]
Jul 17 19:15:51.001227 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:51.001199    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t"
Jul 17 19:15:51.141772 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:51.141695    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fcc3021-0465-4e9a-af54-0de0638d778e-maas-api-service-ca\") pod \"6fcc3021-0465-4e9a-af54-0de0638d778e\" (UID: \"6fcc3021-0465-4e9a-af54-0de0638d778e\") "
Jul 17 19:15:51.142035 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:51.142014    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fcc3021-0465-4e9a-af54-0de0638d778e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6fcc3021-0465-4e9a-af54-0de0638d778e" (UID: "6fcc3021-0465-4e9a-af54-0de0638d778e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:15:51.242873 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:51.242835    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fcc3021-0465-4e9a-af54-0de0638d778e-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:15:51.853420 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:51.853392    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t"
Jul 17 19:15:51.853606 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:51.853434    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t" event={"ID":"6fcc3021-0465-4e9a-af54-0de0638d778e","Type":"ContainerDied","Data":"d4af9567894cac623f2c6493111eb70dc633638dcd2f03b5b1086bb31584ba09"}
Jul 17 19:15:51.853606 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:51.853511    2559 scope.go:117] "RemoveContainer" containerID="b6ac2cbcb5602d6b00573d2b2f7b6841a71fe059ebd28ba3bcf97bf67edbfefb"
Jul 17 19:15:51.874649 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:51.874624    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t"]
Jul 17 19:15:51.880116 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:51.880090    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ghc8t"]
Jul 17 19:15:52.125345 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:52.125255    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fcc3021-0465-4e9a-af54-0de0638d778e" path="/var/lib/kubelet/pods/6fcc3021-0465-4e9a-af54-0de0638d778e/volumes"
Jul 17 19:15:55.605349 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:55.605299    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg"]
Jul 17 19:15:55.605720 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:55.605705    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fcc3021-0465-4e9a-af54-0de0638d778e" containerName="revoke-keys"
Jul 17 19:15:55.605720 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:55.605716    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fcc3021-0465-4e9a-af54-0de0638d778e" containerName="revoke-keys"
Jul 17 19:15:55.605793 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:55.605733    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fcc3021-0465-4e9a-af54-0de0638d778e" containerName="revoke-keys"
Jul 17 19:15:55.605793 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:55.605738    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fcc3021-0465-4e9a-af54-0de0638d778e" containerName="revoke-keys"
Jul 17 19:15:55.605864 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:55.605808    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fcc3021-0465-4e9a-af54-0de0638d778e" containerName="revoke-keys"
Jul 17 19:15:55.605864 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:55.605819    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fcc3021-0465-4e9a-af54-0de0638d778e" containerName="revoke-keys"
Jul 17 19:15:55.605864 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:55.605826    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fcc3021-0465-4e9a-af54-0de0638d778e" containerName="revoke-keys"
Jul 17 19:15:55.610292 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:55.610275    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg"
Jul 17 19:15:55.616028 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:55.616001    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg"]
Jul 17 19:15:55.679882 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:55.679854    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d58ad73e-cdd7-422b-972e-80e6adeacb0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg\" (UID: \"d58ad73e-cdd7-422b-972e-80e6adeacb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg"
Jul 17 19:15:55.781326 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:55.781271    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d58ad73e-cdd7-422b-972e-80e6adeacb0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg\" (UID: \"d58ad73e-cdd7-422b-972e-80e6adeacb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg"
Jul 17 19:15:55.781860 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:55.781841    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d58ad73e-cdd7-422b-972e-80e6adeacb0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg\" (UID: \"d58ad73e-cdd7-422b-972e-80e6adeacb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg"
Jul 17 19:15:55.924126 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:55.924056    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg"
Jul 17 19:15:56.060269 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:56.060237    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg"]
Jul 17 19:15:56.061632 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:15:56.061606    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58ad73e_cdd7_422b_972e_80e6adeacb0e.slice/crio-1582b3ab19f530b1a3c462dccb9f9e7ccbb437a4def1a8509d18aed207dcb649 WatchSource:0}: Error finding container 1582b3ab19f530b1a3c462dccb9f9e7ccbb437a4def1a8509d18aed207dcb649: Status 404 returned error can't find the container with id 1582b3ab19f530b1a3c462dccb9f9e7ccbb437a4def1a8509d18aed207dcb649
Jul 17 19:15:56.120291 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:56.120263    2559 scope.go:117] "RemoveContainer" containerID="996f3408a8d3511afbddb622f168aa1fdc385d4c53d57093e0aa4d8aa2b5e3ae"
Jul 17 19:15:56.884249 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:56.884213    2559 generic.go:356] "Generic (PLEG): container finished" podID="d58ad73e-cdd7-422b-972e-80e6adeacb0e" containerID="0a3279b8c99f9afe5ffaabedaf27188f51c5324b65c013247c65222b2545787a" exitCode=6
Jul 17 19:15:56.884709 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:56.884325    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg" event={"ID":"d58ad73e-cdd7-422b-972e-80e6adeacb0e","Type":"ContainerDied","Data":"0a3279b8c99f9afe5ffaabedaf27188f51c5324b65c013247c65222b2545787a"}
Jul 17 19:15:56.884709 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:56.884389    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg" event={"ID":"d58ad73e-cdd7-422b-972e-80e6adeacb0e","Type":"ContainerStarted","Data":"1582b3ab19f530b1a3c462dccb9f9e7ccbb437a4def1a8509d18aed207dcb649"}
Jul 17 19:15:56.884709 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:56.884546    2559 scope.go:117] "RemoveContainer" containerID="0a3279b8c99f9afe5ffaabedaf27188f51c5324b65c013247c65222b2545787a"
Jul 17 19:15:56.886330 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:56.886289    2559 generic.go:356] "Generic (PLEG): container finished" podID="7880eb14-169a-45fe-add7-0daaf9ad20b5" containerID="6e972e1879f740ea2f34563d838487f8238d8587ff0ea48284974683c0dd6bfb" exitCode=7
Jul 17 19:15:56.886445 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:56.886336    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9" event={"ID":"7880eb14-169a-45fe-add7-0daaf9ad20b5","Type":"ContainerDied","Data":"6e972e1879f740ea2f34563d838487f8238d8587ff0ea48284974683c0dd6bfb"}
Jul 17 19:15:56.886445 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:56.886389    2559 scope.go:117] "RemoveContainer" containerID="996f3408a8d3511afbddb622f168aa1fdc385d4c53d57093e0aa4d8aa2b5e3ae"
Jul 17 19:15:56.886697 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:56.886681    2559 scope.go:117] "RemoveContainer" containerID="6e972e1879f740ea2f34563d838487f8238d8587ff0ea48284974683c0dd6bfb"
Jul 17 19:15:56.886960 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:15:56.886934    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-968j9_odh-ai-gateway-infra(7880eb14-169a-45fe-add7-0daaf9ad20b5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9" podUID="7880eb14-169a-45fe-add7-0daaf9ad20b5"
Jul 17 19:15:57.895108 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:57.895075    2559 generic.go:356] "Generic (PLEG): container finished" podID="d58ad73e-cdd7-422b-972e-80e6adeacb0e" containerID="1a25ba52385f1d32e01ff7dad207bc9b35ab2debb6c7b272c8d26e4f74824005" exitCode=6
Jul 17 19:15:57.895606 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:57.895144    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg" event={"ID":"d58ad73e-cdd7-422b-972e-80e6adeacb0e","Type":"ContainerDied","Data":"1a25ba52385f1d32e01ff7dad207bc9b35ab2debb6c7b272c8d26e4f74824005"}
Jul 17 19:15:57.895606 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:57.895215    2559 scope.go:117] "RemoveContainer" containerID="0a3279b8c99f9afe5ffaabedaf27188f51c5324b65c013247c65222b2545787a"
Jul 17 19:15:57.895606 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:57.895455    2559 scope.go:117] "RemoveContainer" containerID="1a25ba52385f1d32e01ff7dad207bc9b35ab2debb6c7b272c8d26e4f74824005"
Jul 17 19:15:57.895770 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:15:57.895671    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg_odh-ai-gateway-infra(d58ad73e-cdd7-422b-972e-80e6adeacb0e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg" podUID="d58ad73e-cdd7-422b-972e-80e6adeacb0e"
Jul 17 19:15:57.927629 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:57.927608    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9"]
Jul 17 19:15:58.059820 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:58.059799    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9"
Jul 17 19:15:58.204431 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:58.204349    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7880eb14-169a-45fe-add7-0daaf9ad20b5-maas-api-service-ca\") pod \"7880eb14-169a-45fe-add7-0daaf9ad20b5\" (UID: \"7880eb14-169a-45fe-add7-0daaf9ad20b5\") "
Jul 17 19:15:58.204693 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:58.204672    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7880eb14-169a-45fe-add7-0daaf9ad20b5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7880eb14-169a-45fe-add7-0daaf9ad20b5" (UID: "7880eb14-169a-45fe-add7-0daaf9ad20b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:15:58.305735 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:58.305704    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7880eb14-169a-45fe-add7-0daaf9ad20b5-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:15:58.904747 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:58.904714    2559 scope.go:117] "RemoveContainer" containerID="1a25ba52385f1d32e01ff7dad207bc9b35ab2debb6c7b272c8d26e4f74824005"
Jul 17 19:15:58.905205 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:15:58.904941    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg_odh-ai-gateway-infra(d58ad73e-cdd7-422b-972e-80e6adeacb0e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg" podUID="d58ad73e-cdd7-422b-972e-80e6adeacb0e"
Jul 17 19:15:58.905746 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:58.905723    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9" event={"ID":"7880eb14-169a-45fe-add7-0daaf9ad20b5","Type":"ContainerDied","Data":"b2d5cd0834ebd6ecc834b43a6cbec927284bbf567f946d145269a28f6194130a"}
Jul 17 19:15:58.905746 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:58.905738    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9"
Jul 17 19:15:58.905909 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:58.905784    2559 scope.go:117] "RemoveContainer" containerID="6e972e1879f740ea2f34563d838487f8238d8587ff0ea48284974683c0dd6bfb"
Jul 17 19:15:58.932934 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:58.932904    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9"]
Jul 17 19:15:58.946243 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:15:58.940499    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-968j9"]
Jul 17 19:16:00.126010 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:00.125977    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7880eb14-169a-45fe-add7-0daaf9ad20b5" path="/var/lib/kubelet/pods/7880eb14-169a-45fe-add7-0daaf9ad20b5/volumes"
Jul 17 19:16:02.605461 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:02.605426    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv"]
Jul 17 19:16:02.606024 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:02.606003    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fcc3021-0465-4e9a-af54-0de0638d778e" containerName="revoke-keys"
Jul 17 19:16:02.606111 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:02.606027    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fcc3021-0465-4e9a-af54-0de0638d778e" containerName="revoke-keys"
Jul 17 19:16:02.606111 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:02.606045    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7880eb14-169a-45fe-add7-0daaf9ad20b5" containerName="revoke-keys"
Jul 17 19:16:02.606111 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:02.606054    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7880eb14-169a-45fe-add7-0daaf9ad20b5" containerName="revoke-keys"
Jul 17 19:16:02.606111 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:02.606063    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7880eb14-169a-45fe-add7-0daaf9ad20b5" containerName="revoke-keys"
Jul 17 19:16:02.606111 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:02.606072    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7880eb14-169a-45fe-add7-0daaf9ad20b5" containerName="revoke-keys"
Jul 17 19:16:02.606111 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:02.606082    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7880eb14-169a-45fe-add7-0daaf9ad20b5" containerName="revoke-keys"
Jul 17 19:16:02.606111 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:02.606090    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7880eb14-169a-45fe-add7-0daaf9ad20b5" containerName="revoke-keys"
Jul 17 19:16:02.606774 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:02.606204    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7880eb14-169a-45fe-add7-0daaf9ad20b5" containerName="revoke-keys"
Jul 17 19:16:02.606774 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:02.606217    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7880eb14-169a-45fe-add7-0daaf9ad20b5" containerName="revoke-keys"
Jul 17 19:16:02.610964 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:02.610942    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv"
Jul 17 19:16:02.615011 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:02.614987    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv"]
Jul 17 19:16:02.747068 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:02.747030    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1b1ee5-9899-44ab-9b56-66e7667db1c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv\" (UID: \"2d1b1ee5-9899-44ab-9b56-66e7667db1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv"
Jul 17 19:16:02.848060 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:02.848024    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1b1ee5-9899-44ab-9b56-66e7667db1c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv\" (UID: \"2d1b1ee5-9899-44ab-9b56-66e7667db1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv"
Jul 17 19:16:02.848664 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:02.848645    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1b1ee5-9899-44ab-9b56-66e7667db1c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv\" (UID: \"2d1b1ee5-9899-44ab-9b56-66e7667db1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv"
Jul 17 19:16:02.923225 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:02.923147    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv"
Jul 17 19:16:03.064984 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:03.064955    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv"]
Jul 17 19:16:03.066348 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:16:03.066300    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1b1ee5_9899_44ab_9b56_66e7667db1c7.slice/crio-9ea491cc9117397da0823390657cdfdb45447d5943da3f2dc477e47d682135a1 WatchSource:0}: Error finding container 9ea491cc9117397da0823390657cdfdb45447d5943da3f2dc477e47d682135a1: Status 404 returned error can't find the container with id 9ea491cc9117397da0823390657cdfdb45447d5943da3f2dc477e47d682135a1
Jul 17 19:16:03.937677 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:03.937646    2559 generic.go:356] "Generic (PLEG): container finished" podID="2d1b1ee5-9899-44ab-9b56-66e7667db1c7" containerID="80fcb9e4ebce18fcb555af5c4cf97bfacdc1ad582f981b5999efbe84bcb7882b" exitCode=6
Jul 17 19:16:03.938117 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:03.937731    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv" event={"ID":"2d1b1ee5-9899-44ab-9b56-66e7667db1c7","Type":"ContainerDied","Data":"80fcb9e4ebce18fcb555af5c4cf97bfacdc1ad582f981b5999efbe84bcb7882b"}
Jul 17 19:16:03.938117 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:03.937789    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv" event={"ID":"2d1b1ee5-9899-44ab-9b56-66e7667db1c7","Type":"ContainerStarted","Data":"9ea491cc9117397da0823390657cdfdb45447d5943da3f2dc477e47d682135a1"}
Jul 17 19:16:03.938117 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:03.937922    2559 scope.go:117] "RemoveContainer" containerID="80fcb9e4ebce18fcb555af5c4cf97bfacdc1ad582f981b5999efbe84bcb7882b"
Jul 17 19:16:04.944619 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:04.944588    2559 generic.go:356] "Generic (PLEG): container finished" podID="2d1b1ee5-9899-44ab-9b56-66e7667db1c7" containerID="796109a0a1def6a7066ac84b9a8adabdabf9f974a271c6252dbd6c5341f46a97" exitCode=6
Jul 17 19:16:04.945028 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:04.944684    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv" event={"ID":"2d1b1ee5-9899-44ab-9b56-66e7667db1c7","Type":"ContainerDied","Data":"796109a0a1def6a7066ac84b9a8adabdabf9f974a271c6252dbd6c5341f46a97"}
Jul 17 19:16:04.945028 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:04.944745    2559 scope.go:117] "RemoveContainer" containerID="80fcb9e4ebce18fcb555af5c4cf97bfacdc1ad582f981b5999efbe84bcb7882b"
Jul 17 19:16:04.945028 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:04.944905    2559 scope.go:117] "RemoveContainer" containerID="796109a0a1def6a7066ac84b9a8adabdabf9f974a271c6252dbd6c5341f46a97"
Jul 17 19:16:04.945175 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:16:04.945147    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv_odh-ai-gateway-infra(2d1b1ee5-9899-44ab-9b56-66e7667db1c7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv" podUID="2d1b1ee5-9899-44ab-9b56-66e7667db1c7"
Jul 17 19:16:05.952072 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:05.952045    2559 scope.go:117] "RemoveContainer" containerID="796109a0a1def6a7066ac84b9a8adabdabf9f974a271c6252dbd6c5341f46a97"
Jul 17 19:16:05.952486 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:16:05.952249    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv_odh-ai-gateway-infra(2d1b1ee5-9899-44ab-9b56-66e7667db1c7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv" podUID="2d1b1ee5-9899-44ab-9b56-66e7667db1c7"
Jul 17 19:16:13.119696 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:13.119664    2559 scope.go:117] "RemoveContainer" containerID="1a25ba52385f1d32e01ff7dad207bc9b35ab2debb6c7b272c8d26e4f74824005"
Jul 17 19:16:13.996583 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:13.996497    2559 generic.go:356] "Generic (PLEG): container finished" podID="d58ad73e-cdd7-422b-972e-80e6adeacb0e" containerID="1ff57cad85321494ee9e6199073f4709a48f6e9db5c306b4cad0285b975cbb08" exitCode=6
Jul 17 19:16:13.996583 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:13.996568    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg" event={"ID":"d58ad73e-cdd7-422b-972e-80e6adeacb0e","Type":"ContainerDied","Data":"1ff57cad85321494ee9e6199073f4709a48f6e9db5c306b4cad0285b975cbb08"}
Jul 17 19:16:13.996787 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:13.996631    2559 scope.go:117] "RemoveContainer" containerID="1a25ba52385f1d32e01ff7dad207bc9b35ab2debb6c7b272c8d26e4f74824005"
Jul 17 19:16:13.996996 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:13.996965    2559 scope.go:117] "RemoveContainer" containerID="1ff57cad85321494ee9e6199073f4709a48f6e9db5c306b4cad0285b975cbb08"
Jul 17 19:16:13.997220 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:16:13.997197    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg_odh-ai-gateway-infra(d58ad73e-cdd7-422b-972e-80e6adeacb0e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg" podUID="d58ad73e-cdd7-422b-972e-80e6adeacb0e"
Jul 17 19:16:15.022434 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:15.022401    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg"]
Jul 17 19:16:15.160831 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:15.160810    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg"
Jul 17 19:16:15.256692 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:15.256669    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d58ad73e-cdd7-422b-972e-80e6adeacb0e-maas-api-service-ca\") pod \"d58ad73e-cdd7-422b-972e-80e6adeacb0e\" (UID: \"d58ad73e-cdd7-422b-972e-80e6adeacb0e\") "
Jul 17 19:16:15.256977 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:15.256956    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d58ad73e-cdd7-422b-972e-80e6adeacb0e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d58ad73e-cdd7-422b-972e-80e6adeacb0e" (UID: "d58ad73e-cdd7-422b-972e-80e6adeacb0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:16:15.357572 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:15.357548    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d58ad73e-cdd7-422b-972e-80e6adeacb0e-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:16:16.016080 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:16.016052    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg"
Jul 17 19:16:16.016080 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:16.016051    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg" event={"ID":"d58ad73e-cdd7-422b-972e-80e6adeacb0e","Type":"ContainerDied","Data":"1582b3ab19f530b1a3c462dccb9f9e7ccbb437a4def1a8509d18aed207dcb649"}
Jul 17 19:16:16.016292 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:16.016148    2559 scope.go:117] "RemoveContainer" containerID="1ff57cad85321494ee9e6199073f4709a48f6e9db5c306b4cad0285b975cbb08"
Jul 17 19:16:16.038005 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:16.037979    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg"]
Jul 17 19:16:16.043223 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:16.043201    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-ptvcg"]
Jul 17 19:16:16.125532 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:16.125510    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d58ad73e-cdd7-422b-972e-80e6adeacb0e" path="/var/lib/kubelet/pods/d58ad73e-cdd7-422b-972e-80e6adeacb0e/volumes"
Jul 17 19:16:18.119979 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:18.119947    2559 scope.go:117] "RemoveContainer" containerID="796109a0a1def6a7066ac84b9a8adabdabf9f974a271c6252dbd6c5341f46a97"
Jul 17 19:16:19.035463 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:19.035382    2559 generic.go:356] "Generic (PLEG): container finished" podID="2d1b1ee5-9899-44ab-9b56-66e7667db1c7" containerID="592e7c66a0a3dd90aad9b30ce8704ba3804ed22e8371c85801aafbde17a7172b" exitCode=6
Jul 17 19:16:19.035590 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:19.035460    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv" event={"ID":"2d1b1ee5-9899-44ab-9b56-66e7667db1c7","Type":"ContainerDied","Data":"592e7c66a0a3dd90aad9b30ce8704ba3804ed22e8371c85801aafbde17a7172b"}
Jul 17 19:16:19.035590 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:19.035520    2559 scope.go:117] "RemoveContainer" containerID="796109a0a1def6a7066ac84b9a8adabdabf9f974a271c6252dbd6c5341f46a97"
Jul 17 19:16:19.035791 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:19.035775    2559 scope.go:117] "RemoveContainer" containerID="592e7c66a0a3dd90aad9b30ce8704ba3804ed22e8371c85801aafbde17a7172b"
Jul 17 19:16:19.035993 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:16:19.035973    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv_odh-ai-gateway-infra(2d1b1ee5-9899-44ab-9b56-66e7667db1c7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv" podUID="2d1b1ee5-9899-44ab-9b56-66e7667db1c7"
Jul 17 19:16:20.067253 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:20.067225    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv"]
Jul 17 19:16:20.222790 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:20.222764    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv"
Jul 17 19:16:20.292946 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:20.292912    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1b1ee5-9899-44ab-9b56-66e7667db1c7-maas-api-service-ca\") pod \"2d1b1ee5-9899-44ab-9b56-66e7667db1c7\" (UID: \"2d1b1ee5-9899-44ab-9b56-66e7667db1c7\") "
Jul 17 19:16:20.293296 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:20.293271    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1b1ee5-9899-44ab-9b56-66e7667db1c7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d1b1ee5-9899-44ab-9b56-66e7667db1c7" (UID: "2d1b1ee5-9899-44ab-9b56-66e7667db1c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:16:20.394149 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:20.394075    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1b1ee5-9899-44ab-9b56-66e7667db1c7-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:16:21.047808 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:21.047778    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv"
Jul 17 19:16:21.047989 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:21.047819    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv" event={"ID":"2d1b1ee5-9899-44ab-9b56-66e7667db1c7","Type":"ContainerDied","Data":"9ea491cc9117397da0823390657cdfdb45447d5943da3f2dc477e47d682135a1"}
Jul 17 19:16:21.047989 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:21.047882    2559 scope.go:117] "RemoveContainer" containerID="592e7c66a0a3dd90aad9b30ce8704ba3804ed22e8371c85801aafbde17a7172b"
Jul 17 19:16:21.069424 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:21.069401    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv"]
Jul 17 19:16:21.072597 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:21.072576    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-4pqhv"]
Jul 17 19:16:22.125985 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:22.125935    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d1b1ee5-9899-44ab-9b56-66e7667db1c7" path="/var/lib/kubelet/pods/2d1b1ee5-9899-44ab-9b56-66e7667db1c7/volumes"
Jul 17 19:16:23.776174 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.776137    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k"]
Jul 17 19:16:23.776592 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.776519    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1b1ee5-9899-44ab-9b56-66e7667db1c7" containerName="revoke-keys"
Jul 17 19:16:23.776592 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.776530    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1b1ee5-9899-44ab-9b56-66e7667db1c7" containerName="revoke-keys"
Jul 17 19:16:23.776592 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.776538    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d58ad73e-cdd7-422b-972e-80e6adeacb0e" containerName="revoke-keys"
Jul 17 19:16:23.776592 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.776543    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d58ad73e-cdd7-422b-972e-80e6adeacb0e" containerName="revoke-keys"
Jul 17 19:16:23.776592 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.776550    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1b1ee5-9899-44ab-9b56-66e7667db1c7" containerName="revoke-keys"
Jul 17 19:16:23.776592 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.776556    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1b1ee5-9899-44ab-9b56-66e7667db1c7" containerName="revoke-keys"
Jul 17 19:16:23.776592 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.776566    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d58ad73e-cdd7-422b-972e-80e6adeacb0e" containerName="revoke-keys"
Jul 17 19:16:23.776592 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.776571    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d58ad73e-cdd7-422b-972e-80e6adeacb0e" containerName="revoke-keys"
Jul 17 19:16:23.776844 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.776634    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d58ad73e-cdd7-422b-972e-80e6adeacb0e" containerName="revoke-keys"
Jul 17 19:16:23.776844 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.776645    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1b1ee5-9899-44ab-9b56-66e7667db1c7" containerName="revoke-keys"
Jul 17 19:16:23.776844 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.776651    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1b1ee5-9899-44ab-9b56-66e7667db1c7" containerName="revoke-keys"
Jul 17 19:16:23.776844 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.776657    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1b1ee5-9899-44ab-9b56-66e7667db1c7" containerName="revoke-keys"
Jul 17 19:16:23.776844 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.776663    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d58ad73e-cdd7-422b-972e-80e6adeacb0e" containerName="revoke-keys"
Jul 17 19:16:23.776844 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.776668    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7880eb14-169a-45fe-add7-0daaf9ad20b5" containerName="revoke-keys"
Jul 17 19:16:23.781232 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.781213    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k"
Jul 17 19:16:23.784179 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.784156    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:16:23.787236 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.787210    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k"]
Jul 17 19:16:23.823249 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.823226    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87dfc12-f51a-491f-897d-f1f40b169fc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k\" (UID: \"a87dfc12-f51a-491f-897d-f1f40b169fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k"
Jul 17 19:16:23.924347 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.924281    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87dfc12-f51a-491f-897d-f1f40b169fc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k\" (UID: \"a87dfc12-f51a-491f-897d-f1f40b169fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k"
Jul 17 19:16:23.924946 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:23.924919    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87dfc12-f51a-491f-897d-f1f40b169fc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k\" (UID: \"a87dfc12-f51a-491f-897d-f1f40b169fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k"
Jul 17 19:16:24.092516 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:24.092479    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k"
Jul 17 19:16:24.225731 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:24.225705    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k"]
Jul 17 19:16:24.227390 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:16:24.227360    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87dfc12_f51a_491f_897d_f1f40b169fc6.slice/crio-27b1f63d73bbe324162d87622f228fcf53f6468653f94b8da2296ec424d3c203 WatchSource:0}: Error finding container 27b1f63d73bbe324162d87622f228fcf53f6468653f94b8da2296ec424d3c203: Status 404 returned error can't find the container with id 27b1f63d73bbe324162d87622f228fcf53f6468653f94b8da2296ec424d3c203
Jul 17 19:16:25.073282 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:25.073247    2559 generic.go:356] "Generic (PLEG): container finished" podID="a87dfc12-f51a-491f-897d-f1f40b169fc6" containerID="0375822fbc09d7c9ced1f124f95995aed371a799051c65d68fbb88b69023073e" exitCode=6
Jul 17 19:16:25.073282 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:25.073281    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k" event={"ID":"a87dfc12-f51a-491f-897d-f1f40b169fc6","Type":"ContainerDied","Data":"0375822fbc09d7c9ced1f124f95995aed371a799051c65d68fbb88b69023073e"}
Jul 17 19:16:25.073835 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:25.073342    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k" event={"ID":"a87dfc12-f51a-491f-897d-f1f40b169fc6","Type":"ContainerStarted","Data":"27b1f63d73bbe324162d87622f228fcf53f6468653f94b8da2296ec424d3c203"}
Jul 17 19:16:25.073835 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:25.073556    2559 scope.go:117] "RemoveContainer" containerID="0375822fbc09d7c9ced1f124f95995aed371a799051c65d68fbb88b69023073e"
Jul 17 19:16:26.081529 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:26.081496    2559 generic.go:356] "Generic (PLEG): container finished" podID="a87dfc12-f51a-491f-897d-f1f40b169fc6" containerID="0d0513ab702a027afb3186b776a7b76602ede9e640c5997b1730d0a19555be45" exitCode=6
Jul 17 19:16:26.081906 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:26.081563    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k" event={"ID":"a87dfc12-f51a-491f-897d-f1f40b169fc6","Type":"ContainerDied","Data":"0d0513ab702a027afb3186b776a7b76602ede9e640c5997b1730d0a19555be45"}
Jul 17 19:16:26.081906 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:26.081610    2559 scope.go:117] "RemoveContainer" containerID="0375822fbc09d7c9ced1f124f95995aed371a799051c65d68fbb88b69023073e"
Jul 17 19:16:26.081906 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:26.081844    2559 scope.go:117] "RemoveContainer" containerID="0d0513ab702a027afb3186b776a7b76602ede9e640c5997b1730d0a19555be45"
Jul 17 19:16:26.082081 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:16:26.082030    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k_odh-ai-gateway-infra(a87dfc12-f51a-491f-897d-f1f40b169fc6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k" podUID="a87dfc12-f51a-491f-897d-f1f40b169fc6"
Jul 17 19:16:27.088992 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:27.088964    2559 scope.go:117] "RemoveContainer" containerID="0d0513ab702a027afb3186b776a7b76602ede9e640c5997b1730d0a19555be45"
Jul 17 19:16:27.089424 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:16:27.089169    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k_odh-ai-gateway-infra(a87dfc12-f51a-491f-897d-f1f40b169fc6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k" podUID="a87dfc12-f51a-491f-897d-f1f40b169fc6"
Jul 17 19:16:30.921821 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:30.921730    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc"]
Jul 17 19:16:30.922282 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:30.922141    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1b1ee5-9899-44ab-9b56-66e7667db1c7" containerName="revoke-keys"
Jul 17 19:16:30.922282 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:30.922152    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1b1ee5-9899-44ab-9b56-66e7667db1c7" containerName="revoke-keys"
Jul 17 19:16:30.922282 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:30.922163    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d58ad73e-cdd7-422b-972e-80e6adeacb0e" containerName="revoke-keys"
Jul 17 19:16:30.922282 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:30.922168    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d58ad73e-cdd7-422b-972e-80e6adeacb0e" containerName="revoke-keys"
Jul 17 19:16:30.922282 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:30.922223    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d58ad73e-cdd7-422b-972e-80e6adeacb0e" containerName="revoke-keys"
Jul 17 19:16:30.926588 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:30.926569    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc"
Jul 17 19:16:30.931245 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:30.931216    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc"]
Jul 17 19:16:30.985512 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:30.985470    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e4c9e73-f9db-4689-8f09-0cc723dd1bc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-9mslc\" (UID: \"8e4c9e73-f9db-4689-8f09-0cc723dd1bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc"
Jul 17 19:16:31.086826 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:31.086793    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e4c9e73-f9db-4689-8f09-0cc723dd1bc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-9mslc\" (UID: \"8e4c9e73-f9db-4689-8f09-0cc723dd1bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc"
Jul 17 19:16:31.087445 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:31.087424    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e4c9e73-f9db-4689-8f09-0cc723dd1bc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-9mslc\" (UID: \"8e4c9e73-f9db-4689-8f09-0cc723dd1bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc"
Jul 17 19:16:31.240266 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:31.240174    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc"
Jul 17 19:16:31.375386 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:31.375356    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc"]
Jul 17 19:16:31.376971 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:16:31.376942    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4c9e73_f9db_4689_8f09_0cc723dd1bc3.slice/crio-2bfc18897d2d2ab9203b018076bc3c6d9087fe4bda14eeb45674c69dc8ad9338 WatchSource:0}: Error finding container 2bfc18897d2d2ab9203b018076bc3c6d9087fe4bda14eeb45674c69dc8ad9338: Status 404 returned error can't find the container with id 2bfc18897d2d2ab9203b018076bc3c6d9087fe4bda14eeb45674c69dc8ad9338
Jul 17 19:16:32.118475 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:32.118441    2559 generic.go:356] "Generic (PLEG): container finished" podID="8e4c9e73-f9db-4689-8f09-0cc723dd1bc3" containerID="b9250539cd3d286e52e1d684d8537e93ade42d93b62b2390160cf54027baeeb6" exitCode=7
Jul 17 19:16:32.118904 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:32.118487    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc" event={"ID":"8e4c9e73-f9db-4689-8f09-0cc723dd1bc3","Type":"ContainerDied","Data":"b9250539cd3d286e52e1d684d8537e93ade42d93b62b2390160cf54027baeeb6"}
Jul 17 19:16:32.118904 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:32.118524    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc" event={"ID":"8e4c9e73-f9db-4689-8f09-0cc723dd1bc3","Type":"ContainerStarted","Data":"2bfc18897d2d2ab9203b018076bc3c6d9087fe4bda14eeb45674c69dc8ad9338"}
Jul 17 19:16:32.118904 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:32.118713    2559 scope.go:117] "RemoveContainer" containerID="b9250539cd3d286e52e1d684d8537e93ade42d93b62b2390160cf54027baeeb6"
Jul 17 19:16:33.126139 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:33.126107    2559 generic.go:356] "Generic (PLEG): container finished" podID="8e4c9e73-f9db-4689-8f09-0cc723dd1bc3" containerID="4d56f967824f3d40a50b18114cfbc90a2a487c4bb58013f5fef286de64dfeedb" exitCode=7
Jul 17 19:16:33.126580 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:33.126150    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc" event={"ID":"8e4c9e73-f9db-4689-8f09-0cc723dd1bc3","Type":"ContainerDied","Data":"4d56f967824f3d40a50b18114cfbc90a2a487c4bb58013f5fef286de64dfeedb"}
Jul 17 19:16:33.126580 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:33.126192    2559 scope.go:117] "RemoveContainer" containerID="b9250539cd3d286e52e1d684d8537e93ade42d93b62b2390160cf54027baeeb6"
Jul 17 19:16:33.126580 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:33.126384    2559 scope.go:117] "RemoveContainer" containerID="4d56f967824f3d40a50b18114cfbc90a2a487c4bb58013f5fef286de64dfeedb"
Jul 17 19:16:33.126699 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:16:33.126584    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-9mslc_odh-ai-gateway-infra(8e4c9e73-f9db-4689-8f09-0cc723dd1bc3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc" podUID="8e4c9e73-f9db-4689-8f09-0cc723dd1bc3"
Jul 17 19:16:34.132856 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:34.132826    2559 scope.go:117] "RemoveContainer" containerID="4d56f967824f3d40a50b18114cfbc90a2a487c4bb58013f5fef286de64dfeedb"
Jul 17 19:16:34.133350 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:16:34.133068    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-9mslc_odh-ai-gateway-infra(8e4c9e73-f9db-4689-8f09-0cc723dd1bc3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc" podUID="8e4c9e73-f9db-4689-8f09-0cc723dd1bc3"
Jul 17 19:16:41.120299 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:41.120267    2559 scope.go:117] "RemoveContainer" containerID="0d0513ab702a027afb3186b776a7b76602ede9e640c5997b1730d0a19555be45"
Jul 17 19:16:42.179347 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:42.179294    2559 generic.go:356] "Generic (PLEG): container finished" podID="a87dfc12-f51a-491f-897d-f1f40b169fc6" containerID="74b3286cf811d8fd857a9587da169bf6b25ecc620551882d7661b96b3f8833fb" exitCode=6
Jul 17 19:16:42.179744 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:42.179366    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k" event={"ID":"a87dfc12-f51a-491f-897d-f1f40b169fc6","Type":"ContainerDied","Data":"74b3286cf811d8fd857a9587da169bf6b25ecc620551882d7661b96b3f8833fb"}
Jul 17 19:16:42.179744 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:42.179419    2559 scope.go:117] "RemoveContainer" containerID="0d0513ab702a027afb3186b776a7b76602ede9e640c5997b1730d0a19555be45"
Jul 17 19:16:42.179744 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:42.179712    2559 scope.go:117] "RemoveContainer" containerID="74b3286cf811d8fd857a9587da169bf6b25ecc620551882d7661b96b3f8833fb"
Jul 17 19:16:42.179946 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:16:42.179927    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k_odh-ai-gateway-infra(a87dfc12-f51a-491f-897d-f1f40b169fc6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k" podUID="a87dfc12-f51a-491f-897d-f1f40b169fc6"
Jul 17 19:16:43.205500 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:43.205471    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k"]
Jul 17 19:16:43.337206 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:43.337182    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k"
Jul 17 19:16:43.392288 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:43.392259    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87dfc12-f51a-491f-897d-f1f40b169fc6-maas-api-service-ca\") pod \"a87dfc12-f51a-491f-897d-f1f40b169fc6\" (UID: \"a87dfc12-f51a-491f-897d-f1f40b169fc6\") "
Jul 17 19:16:43.392647 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:43.392622    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87dfc12-f51a-491f-897d-f1f40b169fc6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a87dfc12-f51a-491f-897d-f1f40b169fc6" (UID: "a87dfc12-f51a-491f-897d-f1f40b169fc6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:16:43.493820 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:43.493737    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87dfc12-f51a-491f-897d-f1f40b169fc6-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:16:44.192077 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:44.192043    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k" event={"ID":"a87dfc12-f51a-491f-897d-f1f40b169fc6","Type":"ContainerDied","Data":"27b1f63d73bbe324162d87622f228fcf53f6468653f94b8da2296ec424d3c203"}
Jul 17 19:16:44.192077 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:44.192074    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k"
Jul 17 19:16:44.192269 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:44.192111    2559 scope.go:117] "RemoveContainer" containerID="74b3286cf811d8fd857a9587da169bf6b25ecc620551882d7661b96b3f8833fb"
Jul 17 19:16:44.209208 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:44.209186    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k"]
Jul 17 19:16:44.211431 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:44.211411    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-7894k"]
Jul 17 19:16:46.120392 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:46.120368    2559 scope.go:117] "RemoveContainer" containerID="4d56f967824f3d40a50b18114cfbc90a2a487c4bb58013f5fef286de64dfeedb"
Jul 17 19:16:46.126502 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:46.126476    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a87dfc12-f51a-491f-897d-f1f40b169fc6" path="/var/lib/kubelet/pods/a87dfc12-f51a-491f-897d-f1f40b169fc6/volumes"
Jul 17 19:16:47.212321 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:47.212230    2559 generic.go:356] "Generic (PLEG): container finished" podID="8e4c9e73-f9db-4689-8f09-0cc723dd1bc3" containerID="74a713cf42e2918927e7211576a665898d3e8055682924e8155431366333fc90" exitCode=6
Jul 17 19:16:47.212677 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:47.212316    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc" event={"ID":"8e4c9e73-f9db-4689-8f09-0cc723dd1bc3","Type":"ContainerDied","Data":"74a713cf42e2918927e7211576a665898d3e8055682924e8155431366333fc90"}
Jul 17 19:16:47.212677 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:47.212404    2559 scope.go:117] "RemoveContainer" containerID="4d56f967824f3d40a50b18114cfbc90a2a487c4bb58013f5fef286de64dfeedb"
Jul 17 19:16:47.212789 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:47.212681    2559 scope.go:117] "RemoveContainer" containerID="74a713cf42e2918927e7211576a665898d3e8055682924e8155431366333fc90"
Jul 17 19:16:47.212905 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:16:47.212885    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-9mslc_odh-ai-gateway-infra(8e4c9e73-f9db-4689-8f09-0cc723dd1bc3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc" podUID="8e4c9e73-f9db-4689-8f09-0cc723dd1bc3"
Jul 17 19:16:48.237867 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:48.237837    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc"]
Jul 17 19:16:48.377653 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:48.377631    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc"
Jul 17 19:16:48.436153 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:48.436126    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e4c9e73-f9db-4689-8f09-0cc723dd1bc3-maas-api-service-ca\") pod \"8e4c9e73-f9db-4689-8f09-0cc723dd1bc3\" (UID: \"8e4c9e73-f9db-4689-8f09-0cc723dd1bc3\") "
Jul 17 19:16:48.436462 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:48.436442    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4c9e73-f9db-4689-8f09-0cc723dd1bc3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8e4c9e73-f9db-4689-8f09-0cc723dd1bc3" (UID: "8e4c9e73-f9db-4689-8f09-0cc723dd1bc3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:16:48.537888 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:48.537818    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e4c9e73-f9db-4689-8f09-0cc723dd1bc3-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:16:49.225781 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:49.225748    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc" event={"ID":"8e4c9e73-f9db-4689-8f09-0cc723dd1bc3","Type":"ContainerDied","Data":"2bfc18897d2d2ab9203b018076bc3c6d9087fe4bda14eeb45674c69dc8ad9338"}
Jul 17 19:16:49.225781 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:49.225773    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc"
Jul 17 19:16:49.226016 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:49.225803    2559 scope.go:117] "RemoveContainer" containerID="74a713cf42e2918927e7211576a665898d3e8055682924e8155431366333fc90"
Jul 17 19:16:49.250634 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:49.250607    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc"]
Jul 17 19:16:49.252879 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:49.252856    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-9mslc"]
Jul 17 19:16:50.125430 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:16:50.125392    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e4c9e73-f9db-4689-8f09-0cc723dd1bc3" path="/var/lib/kubelet/pods/8e4c9e73-f9db-4689-8f09-0cc723dd1bc3/volumes"
Jul 17 19:18:10.352018 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:10.351935    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 19:18:10.359109 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:10.359085    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 19:18:10.360431 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:10.360415    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 19:18:10.367292 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:10.367275    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 19:18:42.730528 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.730485    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg"]
Jul 17 19:18:42.731140 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.731051    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87dfc12-f51a-491f-897d-f1f40b169fc6" containerName="revoke-keys"
Jul 17 19:18:42.731140 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.731071    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87dfc12-f51a-491f-897d-f1f40b169fc6" containerName="revoke-keys"
Jul 17 19:18:42.731140 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.731087    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e4c9e73-f9db-4689-8f09-0cc723dd1bc3" containerName="revoke-keys"
Jul 17 19:18:42.731140 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.731096    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e4c9e73-f9db-4689-8f09-0cc723dd1bc3" containerName="revoke-keys"
Jul 17 19:18:42.731140 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.731111    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87dfc12-f51a-491f-897d-f1f40b169fc6" containerName="revoke-keys"
Jul 17 19:18:42.731140 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.731120    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87dfc12-f51a-491f-897d-f1f40b169fc6" containerName="revoke-keys"
Jul 17 19:18:42.731496 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.731149    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87dfc12-f51a-491f-897d-f1f40b169fc6" containerName="revoke-keys"
Jul 17 19:18:42.731496 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.731157    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87dfc12-f51a-491f-897d-f1f40b169fc6" containerName="revoke-keys"
Jul 17 19:18:42.731496 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.731166    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e4c9e73-f9db-4689-8f09-0cc723dd1bc3" containerName="revoke-keys"
Jul 17 19:18:42.731496 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.731174    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e4c9e73-f9db-4689-8f09-0cc723dd1bc3" containerName="revoke-keys"
Jul 17 19:18:42.731496 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.731262    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e4c9e73-f9db-4689-8f09-0cc723dd1bc3" containerName="revoke-keys"
Jul 17 19:18:42.731496 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.731273    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e4c9e73-f9db-4689-8f09-0cc723dd1bc3" containerName="revoke-keys"
Jul 17 19:18:42.731496 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.731282    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87dfc12-f51a-491f-897d-f1f40b169fc6" containerName="revoke-keys"
Jul 17 19:18:42.731496 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.731291    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87dfc12-f51a-491f-897d-f1f40b169fc6" containerName="revoke-keys"
Jul 17 19:18:42.734616 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.734595    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg"
Jul 17 19:18:42.738104 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.737926    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:18:42.744271 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.744115    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg"]
Jul 17 19:18:42.813661 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.813617    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40bb7522-1b23-4c6c-9364-35f1288d0d6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg\" (UID: \"40bb7522-1b23-4c6c-9364-35f1288d0d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg"
Jul 17 19:18:42.914807 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.914768    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40bb7522-1b23-4c6c-9364-35f1288d0d6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg\" (UID: \"40bb7522-1b23-4c6c-9364-35f1288d0d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg"
Jul 17 19:18:42.915495 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:42.915470    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40bb7522-1b23-4c6c-9364-35f1288d0d6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg\" (UID: \"40bb7522-1b23-4c6c-9364-35f1288d0d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg"
Jul 17 19:18:43.049147 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:43.049052    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg"
Jul 17 19:18:43.184679 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:43.184652    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg"]
Jul 17 19:18:43.186882 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:18:43.186856    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40bb7522_1b23_4c6c_9364_35f1288d0d6a.slice/crio-c32739465232ac9db2e7a4e3d8f313b0a10193f9499f09b70baebf66e786e56d WatchSource:0}: Error finding container c32739465232ac9db2e7a4e3d8f313b0a10193f9499f09b70baebf66e786e56d: Status 404 returned error can't find the container with id c32739465232ac9db2e7a4e3d8f313b0a10193f9499f09b70baebf66e786e56d
Jul 17 19:18:43.863255 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:43.863222    2559 generic.go:356] "Generic (PLEG): container finished" podID="40bb7522-1b23-4c6c-9364-35f1288d0d6a" containerID="e116f9a4ea0d3baed1453ce034ae2ab957f6ef8ff52b1fea381d995be740dc35" exitCode=6
Jul 17 19:18:43.863673 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:43.863284    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg" event={"ID":"40bb7522-1b23-4c6c-9364-35f1288d0d6a","Type":"ContainerDied","Data":"e116f9a4ea0d3baed1453ce034ae2ab957f6ef8ff52b1fea381d995be740dc35"}
Jul 17 19:18:43.863673 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:43.863348    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg" event={"ID":"40bb7522-1b23-4c6c-9364-35f1288d0d6a","Type":"ContainerStarted","Data":"c32739465232ac9db2e7a4e3d8f313b0a10193f9499f09b70baebf66e786e56d"}
Jul 17 19:18:43.863673 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:43.863509    2559 scope.go:117] "RemoveContainer" containerID="e116f9a4ea0d3baed1453ce034ae2ab957f6ef8ff52b1fea381d995be740dc35"
Jul 17 19:18:44.870690 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:44.870655    2559 generic.go:356] "Generic (PLEG): container finished" podID="40bb7522-1b23-4c6c-9364-35f1288d0d6a" containerID="d59441ce7c81af90eabf2965b104d39d10eda30bea1d8730234cdb8621b8351f" exitCode=6
Jul 17 19:18:44.871107 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:44.870701    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg" event={"ID":"40bb7522-1b23-4c6c-9364-35f1288d0d6a","Type":"ContainerDied","Data":"d59441ce7c81af90eabf2965b104d39d10eda30bea1d8730234cdb8621b8351f"}
Jul 17 19:18:44.871107 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:44.870749    2559 scope.go:117] "RemoveContainer" containerID="e116f9a4ea0d3baed1453ce034ae2ab957f6ef8ff52b1fea381d995be740dc35"
Jul 17 19:18:44.871107 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:44.870952    2559 scope.go:117] "RemoveContainer" containerID="d59441ce7c81af90eabf2965b104d39d10eda30bea1d8730234cdb8621b8351f"
Jul 17 19:18:44.871244 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:18:44.871204    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg_odh-ai-gateway-infra(40bb7522-1b23-4c6c-9364-35f1288d0d6a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg" podUID="40bb7522-1b23-4c6c-9364-35f1288d0d6a"
Jul 17 19:18:44.955845 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:44.955811    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd0f3bb1-pp8qd"]
Jul 17 19:18:44.956222 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:44.956210    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e4c9e73-f9db-4689-8f09-0cc723dd1bc3" containerName="revoke-keys"
Jul 17 19:18:44.956289 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:44.956223    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e4c9e73-f9db-4689-8f09-0cc723dd1bc3" containerName="revoke-keys"
Jul 17 19:18:44.956358 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:44.956330    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e4c9e73-f9db-4689-8f09-0cc723dd1bc3" containerName="revoke-keys"
Jul 17 19:18:44.956358 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:44.956339    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87dfc12-f51a-491f-897d-f1f40b169fc6" containerName="revoke-keys"
Jul 17 19:18:44.960545 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:44.960529    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd0f3bb1-pp8qd"
Jul 17 19:18:44.968971 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:44.968228    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd0f3bb1-pp8qd"]
Jul 17 19:18:45.036627 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:45.036592    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f433815f-03d0-4019-955a-20d4b31fa600-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bd0f3bb1-pp8qd\" (UID: \"f433815f-03d0-4019-955a-20d4b31fa600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd0f3bb1-pp8qd"
Jul 17 19:18:45.137564 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:45.137477    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f433815f-03d0-4019-955a-20d4b31fa600-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bd0f3bb1-pp8qd\" (UID: \"f433815f-03d0-4019-955a-20d4b31fa600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd0f3bb1-pp8qd"
Jul 17 19:18:45.138087 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:45.138068    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f433815f-03d0-4019-955a-20d4b31fa600-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bd0f3bb1-pp8qd\" (UID: \"f433815f-03d0-4019-955a-20d4b31fa600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd0f3bb1-pp8qd"
Jul 17 19:18:45.272703 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:45.272667    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd0f3bb1-pp8qd"
Jul 17 19:18:45.410107 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:45.410073    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd0f3bb1-pp8qd"]
Jul 17 19:18:45.414548 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:18:45.414515    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf433815f_03d0_4019_955a_20d4b31fa600.slice/crio-a799b0ad1f2acbc4326146b73b77bba106c8eaf601a8ecef8800ccbddd3adbbe WatchSource:0}: Error finding container a799b0ad1f2acbc4326146b73b77bba106c8eaf601a8ecef8800ccbddd3adbbe: Status 404 returned error can't find the container with id a799b0ad1f2acbc4326146b73b77bba106c8eaf601a8ecef8800ccbddd3adbbe
Jul 17 19:18:45.878916 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:45.878876    2559 scope.go:117] "RemoveContainer" containerID="d59441ce7c81af90eabf2965b104d39d10eda30bea1d8730234cdb8621b8351f"
Jul 17 19:18:45.879336 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:18:45.879126    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg_odh-ai-gateway-infra(40bb7522-1b23-4c6c-9364-35f1288d0d6a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg" podUID="40bb7522-1b23-4c6c-9364-35f1288d0d6a"
Jul 17 19:18:45.880224 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:45.880201    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd0f3bb1-pp8qd" event={"ID":"f433815f-03d0-4019-955a-20d4b31fa600","Type":"ContainerStarted","Data":"105386538e6c1f675e87b11e97a8e4bb252ae4f3798b987b1e8652c9ba524f40"}
Jul 17 19:18:45.880334 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:45.880243    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd0f3bb1-pp8qd" event={"ID":"f433815f-03d0-4019-955a-20d4b31fa600","Type":"ContainerStarted","Data":"a799b0ad1f2acbc4326146b73b77bba106c8eaf601a8ecef8800ccbddd3adbbe"}
Jul 17 19:18:45.910741 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:45.910700    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd0f3bb1-pp8qd" podStartSLOduration=1.910684402 podStartE2EDuration="1.910684402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:18:45.909924214 +0000 UTC m=+2736.351432902" watchObservedRunningTime="2026-07-17 19:18:45.910684402 +0000 UTC m=+2736.352193093"
Jul 17 19:18:46.895600 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:46.895560    2559 generic.go:356] "Generic (PLEG): container finished" podID="f433815f-03d0-4019-955a-20d4b31fa600" containerID="105386538e6c1f675e87b11e97a8e4bb252ae4f3798b987b1e8652c9ba524f40" exitCode=0
Jul 17 19:18:46.895983 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:46.895647    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd0f3bb1-pp8qd" event={"ID":"f433815f-03d0-4019-955a-20d4b31fa600","Type":"ContainerDied","Data":"105386538e6c1f675e87b11e97a8e4bb252ae4f3798b987b1e8652c9ba524f40"}
Jul 17 19:18:48.032801 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:48.032777    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd0f3bb1-pp8qd"
Jul 17 19:18:48.167618 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:48.167554    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f433815f-03d0-4019-955a-20d4b31fa600-maas-api-service-ca\") pod \"f433815f-03d0-4019-955a-20d4b31fa600\" (UID: \"f433815f-03d0-4019-955a-20d4b31fa600\") "
Jul 17 19:18:48.167870 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:48.167849    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f433815f-03d0-4019-955a-20d4b31fa600-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f433815f-03d0-4019-955a-20d4b31fa600" (UID: "f433815f-03d0-4019-955a-20d4b31fa600"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:18:48.268383 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:48.268359    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f433815f-03d0-4019-955a-20d4b31fa600-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:18:48.907768 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:48.907739    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd0f3bb1-pp8qd"
Jul 17 19:18:48.907941 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:48.907770    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd0f3bb1-pp8qd" event={"ID":"f433815f-03d0-4019-955a-20d4b31fa600","Type":"ContainerDied","Data":"a799b0ad1f2acbc4326146b73b77bba106c8eaf601a8ecef8800ccbddd3adbbe"}
Jul 17 19:18:48.907941 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:18:48.907817    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a799b0ad1f2acbc4326146b73b77bba106c8eaf601a8ecef8800ccbddd3adbbe"
Jul 17 19:19:00.122955 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:00.122924    2559 scope.go:117] "RemoveContainer" containerID="d59441ce7c81af90eabf2965b104d39d10eda30bea1d8730234cdb8621b8351f"
Jul 17 19:19:00.976182 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:00.976151    2559 generic.go:356] "Generic (PLEG): container finished" podID="40bb7522-1b23-4c6c-9364-35f1288d0d6a" containerID="893c163cc34277c37ddabbb0fad603e5d79d30bf2107761e55f31d39166f448b" exitCode=6
Jul 17 19:19:00.976373 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:00.976224    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg" event={"ID":"40bb7522-1b23-4c6c-9364-35f1288d0d6a","Type":"ContainerDied","Data":"893c163cc34277c37ddabbb0fad603e5d79d30bf2107761e55f31d39166f448b"}
Jul 17 19:19:00.976373 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:00.976287    2559 scope.go:117] "RemoveContainer" containerID="d59441ce7c81af90eabf2965b104d39d10eda30bea1d8730234cdb8621b8351f"
Jul 17 19:19:00.976621 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:00.976599    2559 scope.go:117] "RemoveContainer" containerID="893c163cc34277c37ddabbb0fad603e5d79d30bf2107761e55f31d39166f448b"
Jul 17 19:19:00.976846 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:19:00.976821    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg_odh-ai-gateway-infra(40bb7522-1b23-4c6c-9364-35f1288d0d6a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg" podUID="40bb7522-1b23-4c6c-9364-35f1288d0d6a"
Jul 17 19:19:02.012010 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:02.011974    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg"]
Jul 17 19:19:02.149487 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:02.149462    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg"
Jul 17 19:19:02.291647 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:02.291571    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40bb7522-1b23-4c6c-9364-35f1288d0d6a-maas-api-service-ca\") pod \"40bb7522-1b23-4c6c-9364-35f1288d0d6a\" (UID: \"40bb7522-1b23-4c6c-9364-35f1288d0d6a\") "
Jul 17 19:19:02.291889 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:02.291867    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40bb7522-1b23-4c6c-9364-35f1288d0d6a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "40bb7522-1b23-4c6c-9364-35f1288d0d6a" (UID: "40bb7522-1b23-4c6c-9364-35f1288d0d6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:19:02.392505 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:02.392472    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40bb7522-1b23-4c6c-9364-35f1288d0d6a-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:19:02.990195 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:02.990159    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg"
Jul 17 19:19:02.990406 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:02.990192    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg" event={"ID":"40bb7522-1b23-4c6c-9364-35f1288d0d6a","Type":"ContainerDied","Data":"c32739465232ac9db2e7a4e3d8f313b0a10193f9499f09b70baebf66e786e56d"}
Jul 17 19:19:02.990406 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:02.990257    2559 scope.go:117] "RemoveContainer" containerID="893c163cc34277c37ddabbb0fad603e5d79d30bf2107761e55f31d39166f448b"
Jul 17 19:19:03.016940 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:03.016913    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg"]
Jul 17 19:19:03.025248 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:03.025224    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-qvpsg"]
Jul 17 19:19:04.125284 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:04.125252    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40bb7522-1b23-4c6c-9364-35f1288d0d6a" path="/var/lib/kubelet/pods/40bb7522-1b23-4c6c-9364-35f1288d0d6a/volumes"
Jul 17 19:19:11.107331 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:11.107286    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89"]
Jul 17 19:19:11.107709 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:11.107666    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40bb7522-1b23-4c6c-9364-35f1288d0d6a" containerName="revoke-keys"
Jul 17 19:19:11.107709 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:11.107678    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="40bb7522-1b23-4c6c-9364-35f1288d0d6a" containerName="revoke-keys"
Jul 17 19:19:11.107709 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:11.107685    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40bb7522-1b23-4c6c-9364-35f1288d0d6a" containerName="revoke-keys"
Jul 17 19:19:11.107709 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:11.107691    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="40bb7522-1b23-4c6c-9364-35f1288d0d6a" containerName="revoke-keys"
Jul 17 19:19:11.107839 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:11.107716    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f433815f-03d0-4019-955a-20d4b31fa600" containerName="revoke-keys"
Jul 17 19:19:11.107839 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:11.107722    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f433815f-03d0-4019-955a-20d4b31fa600" containerName="revoke-keys"
Jul 17 19:19:11.107839 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:11.107781    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f433815f-03d0-4019-955a-20d4b31fa600" containerName="revoke-keys"
Jul 17 19:19:11.107839 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:11.107789    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="40bb7522-1b23-4c6c-9364-35f1288d0d6a" containerName="revoke-keys"
Jul 17 19:19:11.107839 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:11.107795    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="40bb7522-1b23-4c6c-9364-35f1288d0d6a" containerName="revoke-keys"
Jul 17 19:19:11.112483 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:11.112461    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89"
Jul 17 19:19:11.115332 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:11.115292    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:19:11.125489 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:11.125468    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89"]
Jul 17 19:19:11.260280 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:11.260254    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/235dfa24-6012-41e1-9c33-8edf6d4e0680-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-vwp89\" (UID: \"235dfa24-6012-41e1-9c33-8edf6d4e0680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89"
Jul 17 19:19:11.361084 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:11.361018    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/235dfa24-6012-41e1-9c33-8edf6d4e0680-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-vwp89\" (UID: \"235dfa24-6012-41e1-9c33-8edf6d4e0680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89"
Jul 17 19:19:11.361635 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:11.361618    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/235dfa24-6012-41e1-9c33-8edf6d4e0680-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-vwp89\" (UID: \"235dfa24-6012-41e1-9c33-8edf6d4e0680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89"
Jul 17 19:19:11.423233 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:11.423213    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89"
Jul 17 19:19:11.558566 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:11.558543    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89"]
Jul 17 19:19:11.560165 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:19:11.560138    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235dfa24_6012_41e1_9c33_8edf6d4e0680.slice/crio-23cb7fe5abfc2eae78c16359810f539305178f5ede4b28d01e7fb638687073b2 WatchSource:0}: Error finding container 23cb7fe5abfc2eae78c16359810f539305178f5ede4b28d01e7fb638687073b2: Status 404 returned error can't find the container with id 23cb7fe5abfc2eae78c16359810f539305178f5ede4b28d01e7fb638687073b2
Jul 17 19:19:12.045597 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:12.045510    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89" event={"ID":"235dfa24-6012-41e1-9c33-8edf6d4e0680","Type":"ContainerStarted","Data":"30ec7793d89e3d1943cd6b30da78ac3c2ff7e26ac86813583c90ded516bdd622"}
Jul 17 19:19:12.045597 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:12.045566    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89" event={"ID":"235dfa24-6012-41e1-9c33-8edf6d4e0680","Type":"ContainerStarted","Data":"23cb7fe5abfc2eae78c16359810f539305178f5ede4b28d01e7fb638687073b2"}
Jul 17 19:19:12.063713 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:12.063666    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89" podStartSLOduration=1.063650959 podStartE2EDuration="1.063650959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:19:12.061947728 +0000 UTC m=+2762.503456411" watchObservedRunningTime="2026-07-17 19:19:12.063650959 +0000 UTC m=+2762.505159648"
Jul 17 19:19:13.052996 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:13.052965    2559 generic.go:356] "Generic (PLEG): container finished" podID="235dfa24-6012-41e1-9c33-8edf6d4e0680" containerID="30ec7793d89e3d1943cd6b30da78ac3c2ff7e26ac86813583c90ded516bdd622" exitCode=6
Jul 17 19:19:13.053445 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:13.053048    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89" event={"ID":"235dfa24-6012-41e1-9c33-8edf6d4e0680","Type":"ContainerDied","Data":"30ec7793d89e3d1943cd6b30da78ac3c2ff7e26ac86813583c90ded516bdd622"}
Jul 17 19:19:13.053445 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:13.053205    2559 scope.go:117] "RemoveContainer" containerID="30ec7793d89e3d1943cd6b30da78ac3c2ff7e26ac86813583c90ded516bdd622"
Jul 17 19:19:14.060450 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:14.060420    2559 generic.go:356] "Generic (PLEG): container finished" podID="235dfa24-6012-41e1-9c33-8edf6d4e0680" containerID="30cb52d32e8e78b3715f212bb00c20e9f83e2446177b2ae31d10bc8e92450df2" exitCode=6
Jul 17 19:19:14.060849 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:14.060494    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89" event={"ID":"235dfa24-6012-41e1-9c33-8edf6d4e0680","Type":"ContainerDied","Data":"30cb52d32e8e78b3715f212bb00c20e9f83e2446177b2ae31d10bc8e92450df2"}
Jul 17 19:19:14.060849 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:14.060554    2559 scope.go:117] "RemoveContainer" containerID="30ec7793d89e3d1943cd6b30da78ac3c2ff7e26ac86813583c90ded516bdd622"
Jul 17 19:19:14.060849 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:14.060719    2559 scope.go:117] "RemoveContainer" containerID="30cb52d32e8e78b3715f212bb00c20e9f83e2446177b2ae31d10bc8e92450df2"
Jul 17 19:19:14.060974 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:19:14.060932    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-vwp89_odh-ai-gateway-infra(235dfa24-6012-41e1-9c33-8edf6d4e0680)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89" podUID="235dfa24-6012-41e1-9c33-8edf6d4e0680"
Jul 17 19:19:15.068983 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:15.068942    2559 scope.go:117] "RemoveContainer" containerID="30cb52d32e8e78b3715f212bb00c20e9f83e2446177b2ae31d10bc8e92450df2"
Jul 17 19:19:15.069500 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:19:15.069192    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-vwp89_odh-ai-gateway-infra(235dfa24-6012-41e1-9c33-8edf6d4e0680)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89" podUID="235dfa24-6012-41e1-9c33-8edf6d4e0680"
Jul 17 19:19:27.119791 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:27.119757    2559 scope.go:117] "RemoveContainer" containerID="30cb52d32e8e78b3715f212bb00c20e9f83e2446177b2ae31d10bc8e92450df2"
Jul 17 19:19:28.140741 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:28.140710    2559 generic.go:356] "Generic (PLEG): container finished" podID="235dfa24-6012-41e1-9c33-8edf6d4e0680" containerID="544261119c50dd51aa8a7f68c6f665605fff0ccc3260e5a9e03523a8b38c7b1e" exitCode=6
Jul 17 19:19:28.141207 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:28.140756    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89" event={"ID":"235dfa24-6012-41e1-9c33-8edf6d4e0680","Type":"ContainerDied","Data":"544261119c50dd51aa8a7f68c6f665605fff0ccc3260e5a9e03523a8b38c7b1e"}
Jul 17 19:19:28.141207 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:28.140804    2559 scope.go:117] "RemoveContainer" containerID="30cb52d32e8e78b3715f212bb00c20e9f83e2446177b2ae31d10bc8e92450df2"
Jul 17 19:19:28.141207 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:28.141078    2559 scope.go:117] "RemoveContainer" containerID="544261119c50dd51aa8a7f68c6f665605fff0ccc3260e5a9e03523a8b38c7b1e"
Jul 17 19:19:28.141402 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:19:28.141296    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-vwp89_odh-ai-gateway-infra(235dfa24-6012-41e1-9c33-8edf6d4e0680)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89" podUID="235dfa24-6012-41e1-9c33-8edf6d4e0680"
Jul 17 19:19:29.171644 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:29.171611    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89"]
Jul 17 19:19:29.304440 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:29.304419    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89"
Jul 17 19:19:29.412010 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:29.411971    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/235dfa24-6012-41e1-9c33-8edf6d4e0680-maas-api-service-ca\") pod \"235dfa24-6012-41e1-9c33-8edf6d4e0680\" (UID: \"235dfa24-6012-41e1-9c33-8edf6d4e0680\") "
Jul 17 19:19:29.412345 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:29.412324    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/235dfa24-6012-41e1-9c33-8edf6d4e0680-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "235dfa24-6012-41e1-9c33-8edf6d4e0680" (UID: "235dfa24-6012-41e1-9c33-8edf6d4e0680"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:19:29.513099 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:29.513021    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/235dfa24-6012-41e1-9c33-8edf6d4e0680-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:19:30.153646 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:30.153615    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89" event={"ID":"235dfa24-6012-41e1-9c33-8edf6d4e0680","Type":"ContainerDied","Data":"23cb7fe5abfc2eae78c16359810f539305178f5ede4b28d01e7fb638687073b2"}
Jul 17 19:19:30.153789 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:30.153678    2559 scope.go:117] "RemoveContainer" containerID="544261119c50dd51aa8a7f68c6f665605fff0ccc3260e5a9e03523a8b38c7b1e"
Jul 17 19:19:30.153789 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:30.153629    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89"
Jul 17 19:19:30.170735 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:30.170709    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89"]
Jul 17 19:19:30.173842 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:30.173819    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-vwp89"]
Jul 17 19:19:32.125860 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:32.125823    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="235dfa24-6012-41e1-9c33-8edf6d4e0680" path="/var/lib/kubelet/pods/235dfa24-6012-41e1-9c33-8edf6d4e0680/volumes"
Jul 17 19:19:42.813793 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:42.813752    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m"]
Jul 17 19:19:42.814396 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:42.814353    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="235dfa24-6012-41e1-9c33-8edf6d4e0680" containerName="revoke-keys"
Jul 17 19:19:42.814396 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:42.814373    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="235dfa24-6012-41e1-9c33-8edf6d4e0680" containerName="revoke-keys"
Jul 17 19:19:42.814396 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:42.814395    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40bb7522-1b23-4c6c-9364-35f1288d0d6a" containerName="revoke-keys"
Jul 17 19:19:42.814620 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:42.814403    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="40bb7522-1b23-4c6c-9364-35f1288d0d6a" containerName="revoke-keys"
Jul 17 19:19:42.814620 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:42.814423    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="235dfa24-6012-41e1-9c33-8edf6d4e0680" containerName="revoke-keys"
Jul 17 19:19:42.814620 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:42.814431    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="235dfa24-6012-41e1-9c33-8edf6d4e0680" containerName="revoke-keys"
Jul 17 19:19:42.814620 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:42.814526    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="40bb7522-1b23-4c6c-9364-35f1288d0d6a" containerName="revoke-keys"
Jul 17 19:19:42.814620 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:42.814543    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="235dfa24-6012-41e1-9c33-8edf6d4e0680" containerName="revoke-keys"
Jul 17 19:19:42.814620 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:42.814551    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="235dfa24-6012-41e1-9c33-8edf6d4e0680" containerName="revoke-keys"
Jul 17 19:19:42.819122 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:42.819099    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m"
Jul 17 19:19:42.822065 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:42.822047    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:19:42.828343 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:42.828294    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m"]
Jul 17 19:19:42.936410 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:42.936375    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m\" (UID: \"06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m"
Jul 17 19:19:43.037078 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:43.037040    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m\" (UID: \"06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m"
Jul 17 19:19:43.037804 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:43.037782    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m\" (UID: \"06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m"
Jul 17 19:19:43.130341 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:43.130246    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m"
Jul 17 19:19:43.267856 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:43.267833    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m"]
Jul 17 19:19:43.269885 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:19:43.269859    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06204103_7bdd_4aa3_9bd1_2f69cf8fbcb6.slice/crio-b4c9a4698dfee458c66d9b1b8ec720ed2ad00f6acb960e0d675f2d30b1634343 WatchSource:0}: Error finding container b4c9a4698dfee458c66d9b1b8ec720ed2ad00f6acb960e0d675f2d30b1634343: Status 404 returned error can't find the container with id b4c9a4698dfee458c66d9b1b8ec720ed2ad00f6acb960e0d675f2d30b1634343
Jul 17 19:19:44.237971 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:44.237879    2559 generic.go:356] "Generic (PLEG): container finished" podID="06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6" containerID="4afa7556998b0b9f10be1c9e924736138a1564f45bc8bb0568590c864ada743e" exitCode=6
Jul 17 19:19:44.238359 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:44.237966    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m" event={"ID":"06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6","Type":"ContainerDied","Data":"4afa7556998b0b9f10be1c9e924736138a1564f45bc8bb0568590c864ada743e"}
Jul 17 19:19:44.238359 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:44.238019    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m" event={"ID":"06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6","Type":"ContainerStarted","Data":"b4c9a4698dfee458c66d9b1b8ec720ed2ad00f6acb960e0d675f2d30b1634343"}
Jul 17 19:19:44.238359 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:44.238147    2559 scope.go:117] "RemoveContainer" containerID="4afa7556998b0b9f10be1c9e924736138a1564f45bc8bb0568590c864ada743e"
Jul 17 19:19:45.251979 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:45.251891    2559 generic.go:356] "Generic (PLEG): container finished" podID="06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6" containerID="6d44d1ffbb345ca42e78d7445d9156c485ff7e196aefcf81cb5db95c0e215398" exitCode=6
Jul 17 19:19:45.252385 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:45.251978    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m" event={"ID":"06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6","Type":"ContainerDied","Data":"6d44d1ffbb345ca42e78d7445d9156c485ff7e196aefcf81cb5db95c0e215398"}
Jul 17 19:19:45.252385 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:45.252075    2559 scope.go:117] "RemoveContainer" containerID="4afa7556998b0b9f10be1c9e924736138a1564f45bc8bb0568590c864ada743e"
Jul 17 19:19:45.252385 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:45.252170    2559 scope.go:117] "RemoveContainer" containerID="6d44d1ffbb345ca42e78d7445d9156c485ff7e196aefcf81cb5db95c0e215398"
Jul 17 19:19:45.252385 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:19:45.252376    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m_odh-ai-gateway-infra(06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m" podUID="06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6"
Jul 17 19:19:46.260728 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:46.260704    2559 scope.go:117] "RemoveContainer" containerID="6d44d1ffbb345ca42e78d7445d9156c485ff7e196aefcf81cb5db95c0e215398"
Jul 17 19:19:46.261088 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:19:46.260887    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m_odh-ai-gateway-infra(06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m" podUID="06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6"
Jul 17 19:19:54.105051 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.105009    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"]
Jul 17 19:19:54.105490 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.105423    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="235dfa24-6012-41e1-9c33-8edf6d4e0680" containerName="revoke-keys"
Jul 17 19:19:54.105490 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.105436    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="235dfa24-6012-41e1-9c33-8edf6d4e0680" containerName="revoke-keys"
Jul 17 19:19:54.105564 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.105530    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="235dfa24-6012-41e1-9c33-8edf6d4e0680" containerName="revoke-keys"
Jul 17 19:19:54.108628 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.108613    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:19:54.113200 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.113169    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-010d52ef\"/\"openshift-service-ca.crt\""
Jul 17 19:19:54.114279 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.114257    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-010d52ef\"/\"kube-root-ca.crt\""
Jul 17 19:19:54.114407 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.114258    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-010d52ef\"/\"default-dockercfg-dpc8d\""
Jul 17 19:19:54.114478 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.114428    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-010d52ef\"/\"e2e-lifecycle-model-010d52ef-kserve-self-signed-certs\""
Jul 17 19:19:54.132494 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.132464    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"]
Jul 17 19:19:54.234430 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.234402    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-dshm\") pod \"e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") " pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:19:54.234584 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.234434    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-model-cache\") pod \"e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") " pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:19:54.234584 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.234453    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-tmp-dir\") pod \"e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") " pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:19:54.234672 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.234583    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe5a6fac-30be-47ee-bac3-2bec6f606329-tls-certs\") pod \"e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") " pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:19:54.234672 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.234611    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-home\") pod \"e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") " pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:19:54.234672 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.234632    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dshv2\" (UniqueName: \"kubernetes.io/projected/fe5a6fac-30be-47ee-bac3-2bec6f606329-kube-api-access-dshv2\") pod \"e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") " pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:19:54.335601 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.335573    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe5a6fac-30be-47ee-bac3-2bec6f606329-tls-certs\") pod \"e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") " pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:19:54.335601 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.335602    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-home\") pod \"e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") " pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:19:54.335806 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.335620    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dshv2\" (UniqueName: \"kubernetes.io/projected/fe5a6fac-30be-47ee-bac3-2bec6f606329-kube-api-access-dshv2\") pod \"e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") " pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:19:54.335806 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.335751    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-dshm\") pod \"e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") " pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:19:54.335806 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.335797    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-model-cache\") pod \"e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") " pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:19:54.335965 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.335880    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-tmp-dir\") pod \"e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") " pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:19:54.336047 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.336023    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-home\") pod \"e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") " pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:19:54.336113 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.336046    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-model-cache\") pod \"e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") " pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:19:54.336218 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.336199    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-tmp-dir\") pod \"e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") " pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:19:54.338181 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.338154    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-dshm\") pod \"e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") " pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:19:54.338370 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.338352    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe5a6fac-30be-47ee-bac3-2bec6f606329-tls-certs\") pod \"e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") " pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:19:54.344740 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.344722    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dshv2\" (UniqueName: \"kubernetes.io/projected/fe5a6fac-30be-47ee-bac3-2bec6f606329-kube-api-access-dshv2\") pod \"e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") " pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:19:54.417475 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.417414    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:19:54.555954 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:54.555929    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"]
Jul 17 19:19:54.557801 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:19:54.557768    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe5a6fac_30be_47ee_bac3_2bec6f606329.slice/crio-c1ea1096aff16a0c707b3141364685cef630803d80f0c1147770b8d871dac06d WatchSource:0}: Error finding container c1ea1096aff16a0c707b3141364685cef630803d80f0c1147770b8d871dac06d: Status 404 returned error can't find the container with id c1ea1096aff16a0c707b3141364685cef630803d80f0c1147770b8d871dac06d
Jul 17 19:19:55.311866 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:55.311834    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s" event={"ID":"fe5a6fac-30be-47ee-bac3-2bec6f606329","Type":"ContainerStarted","Data":"e841069b50e4d4923fa329d5b4b28b051ad40e197b91b96bd7c3950bc0c6c293"}
Jul 17 19:19:55.311866 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:55.311875    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s" event={"ID":"fe5a6fac-30be-47ee-bac3-2bec6f606329","Type":"ContainerStarted","Data":"c1ea1096aff16a0c707b3141364685cef630803d80f0c1147770b8d871dac06d"}
Jul 17 19:19:55.331420 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:55.331374    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s" podStartSLOduration=1.331356887 podStartE2EDuration="1.331356887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:19:55.331062457 +0000 UTC m=+2805.772571146" watchObservedRunningTime="2026-07-17 19:19:55.331356887 +0000 UTC m=+2805.772865578"
Jul 17 19:19:57.120485 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:57.120451    2559 scope.go:117] "RemoveContainer" containerID="6d44d1ffbb345ca42e78d7445d9156c485ff7e196aefcf81cb5db95c0e215398"
Jul 17 19:19:58.330912 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:58.330877    2559 generic.go:356] "Generic (PLEG): container finished" podID="06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6" containerID="88d27528af971671e6667a72ddbe2dc31d6b1f81c83284b05b5b6ab27722a1e6" exitCode=6
Jul 17 19:19:58.331419 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:58.330952    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m" event={"ID":"06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6","Type":"ContainerDied","Data":"88d27528af971671e6667a72ddbe2dc31d6b1f81c83284b05b5b6ab27722a1e6"}
Jul 17 19:19:58.331419 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:58.331014    2559 scope.go:117] "RemoveContainer" containerID="6d44d1ffbb345ca42e78d7445d9156c485ff7e196aefcf81cb5db95c0e215398"
Jul 17 19:19:58.331419 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:58.331353    2559 scope.go:117] "RemoveContainer" containerID="88d27528af971671e6667a72ddbe2dc31d6b1f81c83284b05b5b6ab27722a1e6"
Jul 17 19:19:58.331638 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:19:58.331617    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m_odh-ai-gateway-infra(06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m" podUID="06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6"
Jul 17 19:19:59.371804 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:59.371771    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m"]
Jul 17 19:19:59.508167 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:59.508140    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m"
Jul 17 19:19:59.683822 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:59.683744    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6-maas-api-service-ca\") pod \"06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6\" (UID: \"06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6\") "
Jul 17 19:19:59.684081 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:59.684057    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6" (UID: "06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:19:59.784720 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:19:59.784689    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:20:00.344883 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:00.344856    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m"
Jul 17 19:20:00.344883 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:00.344879    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m" event={"ID":"06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6","Type":"ContainerDied","Data":"b4c9a4698dfee458c66d9b1b8ec720ed2ad00f6acb960e0d675f2d30b1634343"}
Jul 17 19:20:00.345128 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:00.344939    2559 scope.go:117] "RemoveContainer" containerID="88d27528af971671e6667a72ddbe2dc31d6b1f81c83284b05b5b6ab27722a1e6"
Jul 17 19:20:00.365960 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:00.365911    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m"]
Jul 17 19:20:00.368328 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:00.368296    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-2l96m"]
Jul 17 19:20:02.125718 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:02.125688    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6" path="/var/lib/kubelet/pods/06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6/volumes"
Jul 17 19:20:04.418366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:04.418332    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:20:04.418756 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:04.418377    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:20:04.431071 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:04.431051    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:20:05.384595 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:05.384566    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:20:18.195344 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:18.195291    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-010d52ef-qbjc2"]
Jul 17 19:20:18.195912 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:18.195841    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6" containerName="revoke-keys"
Jul 17 19:20:18.195912 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:18.195857    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6" containerName="revoke-keys"
Jul 17 19:20:18.195912 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:18.195878    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6" containerName="revoke-keys"
Jul 17 19:20:18.195912 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:18.195883    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6" containerName="revoke-keys"
Jul 17 19:20:18.196124 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:18.195960    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6" containerName="revoke-keys"
Jul 17 19:20:18.196124 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:18.195968    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6" containerName="revoke-keys"
Jul 17 19:20:18.198905 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:18.198890    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-010d52ef-qbjc2"
Jul 17 19:20:18.201425 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:18.201406    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:20:18.207390 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:18.207368    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-010d52ef-qbjc2"]
Jul 17 19:20:18.241748 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:18.241714    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e01a532-d33d-4def-bb24-4b1386a5aa3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-010d52ef-qbjc2\" (UID: \"3e01a532-d33d-4def-bb24-4b1386a5aa3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-010d52ef-qbjc2"
Jul 17 19:20:18.342723 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:18.342672    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e01a532-d33d-4def-bb24-4b1386a5aa3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-010d52ef-qbjc2\" (UID: \"3e01a532-d33d-4def-bb24-4b1386a5aa3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-010d52ef-qbjc2"
Jul 17 19:20:18.343300 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:18.343279    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e01a532-d33d-4def-bb24-4b1386a5aa3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-010d52ef-qbjc2\" (UID: \"3e01a532-d33d-4def-bb24-4b1386a5aa3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-010d52ef-qbjc2"
Jul 17 19:20:18.511287 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:18.511182    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-010d52ef-qbjc2"
Jul 17 19:20:18.652400 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:18.652377    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-010d52ef-qbjc2"]
Jul 17 19:20:18.655264 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:20:18.655233    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e01a532_d33d_4def_bb24_4b1386a5aa3b.slice/crio-85fc211d70a7f8472a73376598aaa3a14d2e187dd6d0dcbc86cec03f8cf74421 WatchSource:0}: Error finding container 85fc211d70a7f8472a73376598aaa3a14d2e187dd6d0dcbc86cec03f8cf74421: Status 404 returned error can't find the container with id 85fc211d70a7f8472a73376598aaa3a14d2e187dd6d0dcbc86cec03f8cf74421
Jul 17 19:20:19.455447 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:19.455407    2559 generic.go:356] "Generic (PLEG): container finished" podID="3e01a532-d33d-4def-bb24-4b1386a5aa3b" containerID="1ca3122e9fc0f09ba085af18a03543bec99144a5a9d6757003f3d3581406d7b3" exitCode=0
Jul 17 19:20:19.455864 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:19.455483    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-010d52ef-qbjc2" event={"ID":"3e01a532-d33d-4def-bb24-4b1386a5aa3b","Type":"ContainerDied","Data":"1ca3122e9fc0f09ba085af18a03543bec99144a5a9d6757003f3d3581406d7b3"}
Jul 17 19:20:19.455864 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:19.455526    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-010d52ef-qbjc2" event={"ID":"3e01a532-d33d-4def-bb24-4b1386a5aa3b","Type":"ContainerStarted","Data":"85fc211d70a7f8472a73376598aaa3a14d2e187dd6d0dcbc86cec03f8cf74421"}
Jul 17 19:20:20.596752 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:20.596730    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-010d52ef-qbjc2"
Jul 17 19:20:20.663735 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:20.663704    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e01a532-d33d-4def-bb24-4b1386a5aa3b-maas-api-service-ca\") pod \"3e01a532-d33d-4def-bb24-4b1386a5aa3b\" (UID: \"3e01a532-d33d-4def-bb24-4b1386a5aa3b\") "
Jul 17 19:20:20.664056 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:20.664034    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e01a532-d33d-4def-bb24-4b1386a5aa3b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3e01a532-d33d-4def-bb24-4b1386a5aa3b" (UID: "3e01a532-d33d-4def-bb24-4b1386a5aa3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:20:20.764524 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:20.764444    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e01a532-d33d-4def-bb24-4b1386a5aa3b-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:20:21.467921 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:21.467894    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-010d52ef-qbjc2"
Jul 17 19:20:21.467921 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:21.467902    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-010d52ef-qbjc2" event={"ID":"3e01a532-d33d-4def-bb24-4b1386a5aa3b","Type":"ContainerDied","Data":"85fc211d70a7f8472a73376598aaa3a14d2e187dd6d0dcbc86cec03f8cf74421"}
Jul 17 19:20:21.468117 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:21.467944    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85fc211d70a7f8472a73376598aaa3a14d2e187dd6d0dcbc86cec03f8cf74421"
Jul 17 19:20:38.276887 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:38.276854    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"]
Jul 17 19:20:53.297386 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.297349    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s" podUID="fe5a6fac-30be-47ee-bac3-2bec6f606329" containerName="main" containerID="cri-o://e841069b50e4d4923fa329d5b4b28b051ad40e197b91b96bd7c3950bc0c6c293" gracePeriod=45
Jul 17 19:20:53.540336 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.540294    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:20:53.547335 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.547302    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dshv2\" (UniqueName: \"kubernetes.io/projected/fe5a6fac-30be-47ee-bac3-2bec6f606329-kube-api-access-dshv2\") pod \"fe5a6fac-30be-47ee-bac3-2bec6f606329\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") "
Jul 17 19:20:53.547473 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.547392    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe5a6fac-30be-47ee-bac3-2bec6f606329-tls-certs\") pod \"fe5a6fac-30be-47ee-bac3-2bec6f606329\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") "
Jul 17 19:20:53.547473 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.547444    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-model-cache\") pod \"fe5a6fac-30be-47ee-bac3-2bec6f606329\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") "
Jul 17 19:20:53.547595 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.547480    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-home\") pod \"fe5a6fac-30be-47ee-bac3-2bec6f606329\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") "
Jul 17 19:20:53.547595 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.547532    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-tmp-dir\") pod \"fe5a6fac-30be-47ee-bac3-2bec6f606329\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") "
Jul 17 19:20:53.547595 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.547573    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-dshm\") pod \"fe5a6fac-30be-47ee-bac3-2bec6f606329\" (UID: \"fe5a6fac-30be-47ee-bac3-2bec6f606329\") "
Jul 17 19:20:53.547743 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.547705    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe5a6fac-30be-47ee-bac3-2bec6f606329" (UID: "fe5a6fac-30be-47ee-bac3-2bec6f606329"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:20:53.547804 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.547749    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-home" (OuterVolumeSpecName: "home") pod "fe5a6fac-30be-47ee-bac3-2bec6f606329" (UID: "fe5a6fac-30be-47ee-bac3-2bec6f606329"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:20:53.547858 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.547815    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe5a6fac-30be-47ee-bac3-2bec6f606329" (UID: "fe5a6fac-30be-47ee-bac3-2bec6f606329"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:20:53.547995 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.547976    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-model-cache\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:20:53.548118 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.548093    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-home\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:20:53.548185 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.548121    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-tmp-dir\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:20:53.549678 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.549656    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe5a6fac-30be-47ee-bac3-2bec6f606329-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe5a6fac-30be-47ee-bac3-2bec6f606329" (UID: "fe5a6fac-30be-47ee-bac3-2bec6f606329"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:20:53.549806 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.549679    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5a6fac-30be-47ee-bac3-2bec6f606329-kube-api-access-dshv2" (OuterVolumeSpecName: "kube-api-access-dshv2") pod "fe5a6fac-30be-47ee-bac3-2bec6f606329" (UID: "fe5a6fac-30be-47ee-bac3-2bec6f606329"). InnerVolumeSpecName "kube-api-access-dshv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:20:53.549806 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.549746    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-dshm" (OuterVolumeSpecName: "dshm") pod "fe5a6fac-30be-47ee-bac3-2bec6f606329" (UID: "fe5a6fac-30be-47ee-bac3-2bec6f606329"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:20:53.648676 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.648647    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dshv2\" (UniqueName: \"kubernetes.io/projected/fe5a6fac-30be-47ee-bac3-2bec6f606329-kube-api-access-dshv2\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:20:53.648676 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.648670    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe5a6fac-30be-47ee-bac3-2bec6f606329-tls-certs\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:20:53.648676 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.648681    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe5a6fac-30be-47ee-bac3-2bec6f606329-dshm\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:20:53.649857 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.649827    2559 generic.go:356] "Generic (PLEG): container finished" podID="fe5a6fac-30be-47ee-bac3-2bec6f606329" containerID="e841069b50e4d4923fa329d5b4b28b051ad40e197b91b96bd7c3950bc0c6c293" exitCode=0
Jul 17 19:20:53.649951 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.649866    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"
Jul 17 19:20:53.649951 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.649916    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s" event={"ID":"fe5a6fac-30be-47ee-bac3-2bec6f606329","Type":"ContainerDied","Data":"e841069b50e4d4923fa329d5b4b28b051ad40e197b91b96bd7c3950bc0c6c293"}
Jul 17 19:20:53.650029 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.649948    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s" event={"ID":"fe5a6fac-30be-47ee-bac3-2bec6f606329","Type":"ContainerDied","Data":"c1ea1096aff16a0c707b3141364685cef630803d80f0c1147770b8d871dac06d"}
Jul 17 19:20:53.650029 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.649966    2559 scope.go:117] "RemoveContainer" containerID="e841069b50e4d4923fa329d5b4b28b051ad40e197b91b96bd7c3950bc0c6c293"
Jul 17 19:20:53.660080 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.660065    2559 scope.go:117] "RemoveContainer" containerID="e841069b50e4d4923fa329d5b4b28b051ad40e197b91b96bd7c3950bc0c6c293"
Jul 17 19:20:53.660370 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:20:53.660352    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e841069b50e4d4923fa329d5b4b28b051ad40e197b91b96bd7c3950bc0c6c293\": container with ID starting with e841069b50e4d4923fa329d5b4b28b051ad40e197b91b96bd7c3950bc0c6c293 not found: ID does not exist" containerID="e841069b50e4d4923fa329d5b4b28b051ad40e197b91b96bd7c3950bc0c6c293"
Jul 17 19:20:53.660443 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.660383    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e841069b50e4d4923fa329d5b4b28b051ad40e197b91b96bd7c3950bc0c6c293"} err="failed to get container status \"e841069b50e4d4923fa329d5b4b28b051ad40e197b91b96bd7c3950bc0c6c293\": rpc error: code = NotFound desc = could not find container \"e841069b50e4d4923fa329d5b4b28b051ad40e197b91b96bd7c3950bc0c6c293\": container with ID starting with e841069b50e4d4923fa329d5b4b28b051ad40e197b91b96bd7c3950bc0c6c293 not found: ID does not exist"
Jul 17 19:20:53.673047 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.673025    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"]
Jul 17 19:20:53.678451 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:53.678428    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-010d52ef/e2e-lifecycle-model-010d52ef-kserve-6bfc975447-bf64s"]
Jul 17 19:20:54.126148 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:20:54.126112    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe5a6fac-30be-47ee-bac3-2bec6f606329" path="/var/lib/kubelet/pods/fe5a6fac-30be-47ee-bac3-2bec6f606329/volumes"
Jul 17 19:21:14.019799 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.019760    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9"]
Jul 17 19:21:14.020209 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.020184    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe5a6fac-30be-47ee-bac3-2bec6f606329" containerName="main"
Jul 17 19:21:14.020209 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.020196    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe5a6fac-30be-47ee-bac3-2bec6f606329" containerName="main"
Jul 17 19:21:14.020209 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.020210    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6" containerName="revoke-keys"
Jul 17 19:21:14.020363 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.020216    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6" containerName="revoke-keys"
Jul 17 19:21:14.020363 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.020222    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e01a532-d33d-4def-bb24-4b1386a5aa3b" containerName="revoke-keys"
Jul 17 19:21:14.020363 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.020227    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e01a532-d33d-4def-bb24-4b1386a5aa3b" containerName="revoke-keys"
Jul 17 19:21:14.020363 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.020285    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe5a6fac-30be-47ee-bac3-2bec6f606329" containerName="main"
Jul 17 19:21:14.020363 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.020294    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="06204103-7bdd-4aa3-9bd1-2f69cf8fbcb6" containerName="revoke-keys"
Jul 17 19:21:14.020363 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.020300    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e01a532-d33d-4def-bb24-4b1386a5aa3b" containerName="revoke-keys"
Jul 17 19:21:14.024980 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.024964    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9"
Jul 17 19:21:14.027871 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.027843    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:21:14.031376 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.031350    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9"]
Jul 17 19:21:14.104530 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.104504    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76d2decc-f948-4fb8-975d-661054f85d4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9\" (UID: \"76d2decc-f948-4fb8-975d-661054f85d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9"
Jul 17 19:21:14.204963 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.204934    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76d2decc-f948-4fb8-975d-661054f85d4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9\" (UID: \"76d2decc-f948-4fb8-975d-661054f85d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9"
Jul 17 19:21:14.205566 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.205545    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76d2decc-f948-4fb8-975d-661054f85d4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9\" (UID: \"76d2decc-f948-4fb8-975d-661054f85d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9"
Jul 17 19:21:14.337211 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.337177    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9"
Jul 17 19:21:14.487023 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.486998    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9"]
Jul 17 19:21:14.777738 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.777703    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9" event={"ID":"76d2decc-f948-4fb8-975d-661054f85d4a","Type":"ContainerStarted","Data":"05a683b2e0db9a9703de2284771450648afb112ad607bccb72f006f9783f564e"}
Jul 17 19:21:14.777856 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.777762    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9" event={"ID":"76d2decc-f948-4fb8-975d-661054f85d4a","Type":"ContainerStarted","Data":"af083b398156e09bb325f161705a9cf67c3c845e9c4907f52074a3797932c3a5"}
Jul 17 19:21:14.794856 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:14.794799    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9" podStartSLOduration=0.794783532 podStartE2EDuration="794.783532ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:21:14.793370935 +0000 UTC m=+2885.234879623" watchObservedRunningTime="2026-07-17 19:21:14.794783532 +0000 UTC m=+2885.236292221"
Jul 17 19:21:15.785442 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:15.785406    2559 generic.go:356] "Generic (PLEG): container finished" podID="76d2decc-f948-4fb8-975d-661054f85d4a" containerID="05a683b2e0db9a9703de2284771450648afb112ad607bccb72f006f9783f564e" exitCode=6
Jul 17 19:21:15.785901 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:15.785494    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9" event={"ID":"76d2decc-f948-4fb8-975d-661054f85d4a","Type":"ContainerDied","Data":"05a683b2e0db9a9703de2284771450648afb112ad607bccb72f006f9783f564e"}
Jul 17 19:21:15.785901 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:15.785695    2559 scope.go:117] "RemoveContainer" containerID="05a683b2e0db9a9703de2284771450648afb112ad607bccb72f006f9783f564e"
Jul 17 19:21:16.792803 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:16.792703    2559 generic.go:356] "Generic (PLEG): container finished" podID="76d2decc-f948-4fb8-975d-661054f85d4a" containerID="de8e0f5bf9a10beb23d3235306ba072e73038fd68c9c920efe1b329ead2a89a0" exitCode=6
Jul 17 19:21:16.792803 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:16.792788    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9" event={"ID":"76d2decc-f948-4fb8-975d-661054f85d4a","Type":"ContainerDied","Data":"de8e0f5bf9a10beb23d3235306ba072e73038fd68c9c920efe1b329ead2a89a0"}
Jul 17 19:21:16.793271 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:16.792860    2559 scope.go:117] "RemoveContainer" containerID="05a683b2e0db9a9703de2284771450648afb112ad607bccb72f006f9783f564e"
Jul 17 19:21:16.793271 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:16.793037    2559 scope.go:117] "RemoveContainer" containerID="de8e0f5bf9a10beb23d3235306ba072e73038fd68c9c920efe1b329ead2a89a0"
Jul 17 19:21:16.793405 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:21:16.793286    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9_odh-ai-gateway-infra(76d2decc-f948-4fb8-975d-661054f85d4a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9" podUID="76d2decc-f948-4fb8-975d-661054f85d4a"
Jul 17 19:21:17.800438 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:17.800415    2559 scope.go:117] "RemoveContainer" containerID="de8e0f5bf9a10beb23d3235306ba072e73038fd68c9c920efe1b329ead2a89a0"
Jul 17 19:21:17.800907 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:21:17.800632    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9_odh-ai-gateway-infra(76d2decc-f948-4fb8-975d-661054f85d4a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9" podUID="76d2decc-f948-4fb8-975d-661054f85d4a"
Jul 17 19:21:30.122716 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:30.122680    2559 scope.go:117] "RemoveContainer" containerID="de8e0f5bf9a10beb23d3235306ba072e73038fd68c9c920efe1b329ead2a89a0"
Jul 17 19:21:30.874138 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:30.874106    2559 generic.go:356] "Generic (PLEG): container finished" podID="76d2decc-f948-4fb8-975d-661054f85d4a" containerID="a74e9db63e55c10c340a4c0170ca3f7ab5e229900268e5e86cc858e9cf7f4880" exitCode=6
Jul 17 19:21:30.874331 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:30.874175    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9" event={"ID":"76d2decc-f948-4fb8-975d-661054f85d4a","Type":"ContainerDied","Data":"a74e9db63e55c10c340a4c0170ca3f7ab5e229900268e5e86cc858e9cf7f4880"}
Jul 17 19:21:30.874331 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:30.874243    2559 scope.go:117] "RemoveContainer" containerID="de8e0f5bf9a10beb23d3235306ba072e73038fd68c9c920efe1b329ead2a89a0"
Jul 17 19:21:30.874545 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:30.874530    2559 scope.go:117] "RemoveContainer" containerID="a74e9db63e55c10c340a4c0170ca3f7ab5e229900268e5e86cc858e9cf7f4880"
Jul 17 19:21:30.874801 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:21:30.874782    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9_odh-ai-gateway-infra(76d2decc-f948-4fb8-975d-661054f85d4a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9" podUID="76d2decc-f948-4fb8-975d-661054f85d4a"
Jul 17 19:21:31.902000 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:31.901964    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9"]
Jul 17 19:21:32.040863 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:32.040832    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9"
Jul 17 19:21:32.166542 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:32.166447    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76d2decc-f948-4fb8-975d-661054f85d4a-maas-api-service-ca\") pod \"76d2decc-f948-4fb8-975d-661054f85d4a\" (UID: \"76d2decc-f948-4fb8-975d-661054f85d4a\") "
Jul 17 19:21:32.166802 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:32.166780    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76d2decc-f948-4fb8-975d-661054f85d4a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "76d2decc-f948-4fb8-975d-661054f85d4a" (UID: "76d2decc-f948-4fb8-975d-661054f85d4a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:21:32.267629 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:32.267584    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76d2decc-f948-4fb8-975d-661054f85d4a-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:21:32.888690 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:32.888656    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9"
Jul 17 19:21:32.888904 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:32.888699    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9" event={"ID":"76d2decc-f948-4fb8-975d-661054f85d4a","Type":"ContainerDied","Data":"af083b398156e09bb325f161705a9cf67c3c845e9c4907f52074a3797932c3a5"}
Jul 17 19:21:32.888904 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:32.888768    2559 scope.go:117] "RemoveContainer" containerID="a74e9db63e55c10c340a4c0170ca3f7ab5e229900268e5e86cc858e9cf7f4880"
Jul 17 19:21:32.933114 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:32.933078    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv"]
Jul 17 19:21:32.934461 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:32.934434    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76d2decc-f948-4fb8-975d-661054f85d4a" containerName="revoke-keys"
Jul 17 19:21:32.934612 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:32.934601    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="76d2decc-f948-4fb8-975d-661054f85d4a" containerName="revoke-keys"
Jul 17 19:21:32.934706 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:32.934697    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76d2decc-f948-4fb8-975d-661054f85d4a" containerName="revoke-keys"
Jul 17 19:21:32.934778 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:32.934770    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="76d2decc-f948-4fb8-975d-661054f85d4a" containerName="revoke-keys"
Jul 17 19:21:32.935068 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:32.935055    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="76d2decc-f948-4fb8-975d-661054f85d4a" containerName="revoke-keys"
Jul 17 19:21:32.935173 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:32.935163    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="76d2decc-f948-4fb8-975d-661054f85d4a" containerName="revoke-keys"
Jul 17 19:21:32.940965 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:32.940934    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9"]
Jul 17 19:21:32.941076 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:32.941020    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv"
Jul 17 19:21:32.943507 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:32.943488    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:21:32.950296 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:32.950270    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv"]
Jul 17 19:21:32.953526 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:32.953505    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-q6wp9"]
Jul 17 19:21:33.074606 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:33.074568    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b701b59-1545-4954-997e-d07611d44d0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv\" (UID: \"6b701b59-1545-4954-997e-d07611d44d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv"
Jul 17 19:21:33.176118 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:33.176015    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b701b59-1545-4954-997e-d07611d44d0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv\" (UID: \"6b701b59-1545-4954-997e-d07611d44d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv"
Jul 17 19:21:33.176683 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:33.176661    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b701b59-1545-4954-997e-d07611d44d0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv\" (UID: \"6b701b59-1545-4954-997e-d07611d44d0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv"
Jul 17 19:21:33.253209 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:33.253176    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv"
Jul 17 19:21:33.387039 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:33.387015    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv"]
Jul 17 19:21:33.388584 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:21:33.388547    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b701b59_1545_4954_997e_d07611d44d0b.slice/crio-706f9c811c9ad49d0658157e95246737c35d764b42fe3412affc5358b8abb994 WatchSource:0}: Error finding container 706f9c811c9ad49d0658157e95246737c35d764b42fe3412affc5358b8abb994: Status 404 returned error can't find the container with id 706f9c811c9ad49d0658157e95246737c35d764b42fe3412affc5358b8abb994
Jul 17 19:21:33.896151 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:33.896111    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv" event={"ID":"6b701b59-1545-4954-997e-d07611d44d0b","Type":"ContainerStarted","Data":"c68ab840dcf7970598fed0b88878153a2e5a00bcbe5e326ff004a1fd31504273"}
Jul 17 19:21:33.896397 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:33.896177    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv" event={"ID":"6b701b59-1545-4954-997e-d07611d44d0b","Type":"ContainerStarted","Data":"706f9c811c9ad49d0658157e95246737c35d764b42fe3412affc5358b8abb994"}
Jul 17 19:21:33.912777 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:33.912716    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv" podStartSLOduration=1.912699503 podStartE2EDuration="1.912699503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:21:33.910403347 +0000 UTC m=+2904.351912042" watchObservedRunningTime="2026-07-17 19:21:33.912699503 +0000 UTC m=+2904.354208191"
Jul 17 19:21:34.125758 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:34.125723    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76d2decc-f948-4fb8-975d-661054f85d4a" path="/var/lib/kubelet/pods/76d2decc-f948-4fb8-975d-661054f85d4a/volumes"
Jul 17 19:21:34.904273 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:34.904237    2559 generic.go:356] "Generic (PLEG): container finished" podID="6b701b59-1545-4954-997e-d07611d44d0b" containerID="c68ab840dcf7970598fed0b88878153a2e5a00bcbe5e326ff004a1fd31504273" exitCode=6
Jul 17 19:21:34.904514 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:34.904296    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv" event={"ID":"6b701b59-1545-4954-997e-d07611d44d0b","Type":"ContainerDied","Data":"c68ab840dcf7970598fed0b88878153a2e5a00bcbe5e326ff004a1fd31504273"}
Jul 17 19:21:34.904595 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:34.904536    2559 scope.go:117] "RemoveContainer" containerID="c68ab840dcf7970598fed0b88878153a2e5a00bcbe5e326ff004a1fd31504273"
Jul 17 19:21:35.912052 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:35.912015    2559 generic.go:356] "Generic (PLEG): container finished" podID="6b701b59-1545-4954-997e-d07611d44d0b" containerID="18642ffdfe7457dec3072584aed50b14f6d6df65c260ec9d9466b8c46771b3e0" exitCode=6
Jul 17 19:21:35.912587 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:35.912080    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv" event={"ID":"6b701b59-1545-4954-997e-d07611d44d0b","Type":"ContainerDied","Data":"18642ffdfe7457dec3072584aed50b14f6d6df65c260ec9d9466b8c46771b3e0"}
Jul 17 19:21:35.912587 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:35.912136    2559 scope.go:117] "RemoveContainer" containerID="c68ab840dcf7970598fed0b88878153a2e5a00bcbe5e326ff004a1fd31504273"
Jul 17 19:21:35.912587 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:35.912370    2559 scope.go:117] "RemoveContainer" containerID="18642ffdfe7457dec3072584aed50b14f6d6df65c260ec9d9466b8c46771b3e0"
Jul 17 19:21:35.912752 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:21:35.912639    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv_odh-ai-gateway-infra(6b701b59-1545-4954-997e-d07611d44d0b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv" podUID="6b701b59-1545-4954-997e-d07611d44d0b"
Jul 17 19:21:36.919064 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:36.919032    2559 scope.go:117] "RemoveContainer" containerID="18642ffdfe7457dec3072584aed50b14f6d6df65c260ec9d9466b8c46771b3e0"
Jul 17 19:21:36.919552 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:21:36.919282    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv_odh-ai-gateway-infra(6b701b59-1545-4954-997e-d07611d44d0b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv" podUID="6b701b59-1545-4954-997e-d07611d44d0b"
Jul 17 19:21:50.122895 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:50.122861    2559 scope.go:117] "RemoveContainer" containerID="18642ffdfe7457dec3072584aed50b14f6d6df65c260ec9d9466b8c46771b3e0"
Jul 17 19:21:50.999097 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:50.998998    2559 generic.go:356] "Generic (PLEG): container finished" podID="6b701b59-1545-4954-997e-d07611d44d0b" containerID="32521a7fd7b3465d576105439b7f89673df37027d2186a5f4c239f0715d99881" exitCode=6
Jul 17 19:21:50.999097 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:50.999078    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv" event={"ID":"6b701b59-1545-4954-997e-d07611d44d0b","Type":"ContainerDied","Data":"32521a7fd7b3465d576105439b7f89673df37027d2186a5f4c239f0715d99881"}
Jul 17 19:21:50.999344 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:50.999143    2559 scope.go:117] "RemoveContainer" containerID="18642ffdfe7457dec3072584aed50b14f6d6df65c260ec9d9466b8c46771b3e0"
Jul 17 19:21:50.999551 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:50.999534    2559 scope.go:117] "RemoveContainer" containerID="32521a7fd7b3465d576105439b7f89673df37027d2186a5f4c239f0715d99881"
Jul 17 19:21:50.999783 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:21:50.999763    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv_odh-ai-gateway-infra(6b701b59-1545-4954-997e-d07611d44d0b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv" podUID="6b701b59-1545-4954-997e-d07611d44d0b"
Jul 17 19:21:52.026942 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:52.026901    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv"]
Jul 17 19:21:52.164915 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:52.164887    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv"
Jul 17 19:21:52.261120 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:52.261088    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b701b59-1545-4954-997e-d07611d44d0b-maas-api-service-ca\") pod \"6b701b59-1545-4954-997e-d07611d44d0b\" (UID: \"6b701b59-1545-4954-997e-d07611d44d0b\") "
Jul 17 19:21:52.261513 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:52.261491    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b701b59-1545-4954-997e-d07611d44d0b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6b701b59-1545-4954-997e-d07611d44d0b" (UID: "6b701b59-1545-4954-997e-d07611d44d0b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:21:52.362025 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:52.361994    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b701b59-1545-4954-997e-d07611d44d0b-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:21:53.013856 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:53.013815    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv" event={"ID":"6b701b59-1545-4954-997e-d07611d44d0b","Type":"ContainerDied","Data":"706f9c811c9ad49d0658157e95246737c35d764b42fe3412affc5358b8abb994"}
Jul 17 19:21:53.014040 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:53.013874    2559 scope.go:117] "RemoveContainer" containerID="32521a7fd7b3465d576105439b7f89673df37027d2186a5f4c239f0715d99881"
Jul 17 19:21:53.014040 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:53.013834    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv"
Jul 17 19:21:53.039604 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:53.039581    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv"]
Jul 17 19:21:53.046336 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:53.043551    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7wzkv"]
Jul 17 19:21:54.125452 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:21:54.125418    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b701b59-1545-4954-997e-d07611d44d0b" path="/var/lib/kubelet/pods/6b701b59-1545-4954-997e-d07611d44d0b/volumes"
Jul 17 19:22:45.914069 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:45.913991    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b"]
Jul 17 19:22:45.914696 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:45.914459    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b701b59-1545-4954-997e-d07611d44d0b" containerName="revoke-keys"
Jul 17 19:22:45.914696 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:45.914474    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b701b59-1545-4954-997e-d07611d44d0b" containerName="revoke-keys"
Jul 17 19:22:45.914696 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:45.914491    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b701b59-1545-4954-997e-d07611d44d0b" containerName="revoke-keys"
Jul 17 19:22:45.914696 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:45.914501    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b701b59-1545-4954-997e-d07611d44d0b" containerName="revoke-keys"
Jul 17 19:22:45.914696 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:45.914522    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76d2decc-f948-4fb8-975d-661054f85d4a" containerName="revoke-keys"
Jul 17 19:22:45.914696 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:45.914528    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="76d2decc-f948-4fb8-975d-661054f85d4a" containerName="revoke-keys"
Jul 17 19:22:45.914696 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:45.914605    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b701b59-1545-4954-997e-d07611d44d0b" containerName="revoke-keys"
Jul 17 19:22:45.914696 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:45.914613    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="76d2decc-f948-4fb8-975d-661054f85d4a" containerName="revoke-keys"
Jul 17 19:22:45.914696 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:45.914621    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b701b59-1545-4954-997e-d07611d44d0b" containerName="revoke-keys"
Jul 17 19:22:45.914696 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:45.914627    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b701b59-1545-4954-997e-d07611d44d0b" containerName="revoke-keys"
Jul 17 19:22:45.918057 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:45.918041    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b"
Jul 17 19:22:45.920630 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:45.920610    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:22:45.925510 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:45.925487    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b"]
Jul 17 19:22:45.944317 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:45.944285    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/467e2d3b-ee06-4089-9aa7-0602eb480cfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b\" (UID: \"467e2d3b-ee06-4089-9aa7-0602eb480cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b"
Jul 17 19:22:46.045795 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:46.045759    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/467e2d3b-ee06-4089-9aa7-0602eb480cfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b\" (UID: \"467e2d3b-ee06-4089-9aa7-0602eb480cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b"
Jul 17 19:22:46.046449 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:46.046425    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/467e2d3b-ee06-4089-9aa7-0602eb480cfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b\" (UID: \"467e2d3b-ee06-4089-9aa7-0602eb480cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b"
Jul 17 19:22:46.230242 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:46.230146    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b"
Jul 17 19:22:46.373182 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:46.373158    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b"]
Jul 17 19:22:46.374823 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:22:46.374763    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467e2d3b_ee06_4089_9aa7_0602eb480cfc.slice/crio-8272f2bf4d42fc14e61cc147a28e3e2f9150fa10b5b856f9e702688938ee957e WatchSource:0}: Error finding container 8272f2bf4d42fc14e61cc147a28e3e2f9150fa10b5b856f9e702688938ee957e: Status 404 returned error can't find the container with id 8272f2bf4d42fc14e61cc147a28e3e2f9150fa10b5b856f9e702688938ee957e
Jul 17 19:22:47.318590 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:47.318562    2559 generic.go:356] "Generic (PLEG): container finished" podID="467e2d3b-ee06-4089-9aa7-0602eb480cfc" containerID="f9d87e577d92748dbc2e58a5a633fabab5809558e04b2f143e51097df293054b" exitCode=6
Jul 17 19:22:47.319031 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:47.318645    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b" event={"ID":"467e2d3b-ee06-4089-9aa7-0602eb480cfc","Type":"ContainerDied","Data":"f9d87e577d92748dbc2e58a5a633fabab5809558e04b2f143e51097df293054b"}
Jul 17 19:22:47.319031 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:47.318707    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b" event={"ID":"467e2d3b-ee06-4089-9aa7-0602eb480cfc","Type":"ContainerStarted","Data":"8272f2bf4d42fc14e61cc147a28e3e2f9150fa10b5b856f9e702688938ee957e"}
Jul 17 19:22:47.319031 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:47.318825    2559 scope.go:117] "RemoveContainer" containerID="f9d87e577d92748dbc2e58a5a633fabab5809558e04b2f143e51097df293054b"
Jul 17 19:22:48.326568 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:48.326534    2559 generic.go:356] "Generic (PLEG): container finished" podID="467e2d3b-ee06-4089-9aa7-0602eb480cfc" containerID="72bcca5f6edea22dd7bc8318848492ff223d62c9a88099a01465c0e562bebd82" exitCode=6
Jul 17 19:22:48.327028 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:48.326610    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b" event={"ID":"467e2d3b-ee06-4089-9aa7-0602eb480cfc","Type":"ContainerDied","Data":"72bcca5f6edea22dd7bc8318848492ff223d62c9a88099a01465c0e562bebd82"}
Jul 17 19:22:48.327028 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:48.326675    2559 scope.go:117] "RemoveContainer" containerID="f9d87e577d92748dbc2e58a5a633fabab5809558e04b2f143e51097df293054b"
Jul 17 19:22:48.327028 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:48.326898    2559 scope.go:117] "RemoveContainer" containerID="72bcca5f6edea22dd7bc8318848492ff223d62c9a88099a01465c0e562bebd82"
Jul 17 19:22:48.327196 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:22:48.327119    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b_odh-ai-gateway-infra(467e2d3b-ee06-4089-9aa7-0602eb480cfc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b" podUID="467e2d3b-ee06-4089-9aa7-0602eb480cfc"
Jul 17 19:22:49.335064 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:49.335032    2559 scope.go:117] "RemoveContainer" containerID="72bcca5f6edea22dd7bc8318848492ff223d62c9a88099a01465c0e562bebd82"
Jul 17 19:22:49.335467 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:22:49.335279    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b_odh-ai-gateway-infra(467e2d3b-ee06-4089-9aa7-0602eb480cfc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b" podUID="467e2d3b-ee06-4089-9aa7-0602eb480cfc"
Jul 17 19:22:54.113027 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:54.112989    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h"]
Jul 17 19:22:54.113540 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:54.113429    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b701b59-1545-4954-997e-d07611d44d0b" containerName="revoke-keys"
Jul 17 19:22:54.113540 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:54.113441    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b701b59-1545-4954-997e-d07611d44d0b" containerName="revoke-keys"
Jul 17 19:22:54.117729 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:54.117713    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h"
Jul 17 19:22:54.128110 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:54.128083    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h"]
Jul 17 19:22:54.223776 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:54.223741    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fc88b47-19fc-4246-9b30-5cde5024fc76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h\" (UID: \"4fc88b47-19fc-4246-9b30-5cde5024fc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h"
Jul 17 19:22:54.324974 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:54.324943    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fc88b47-19fc-4246-9b30-5cde5024fc76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h\" (UID: \"4fc88b47-19fc-4246-9b30-5cde5024fc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h"
Jul 17 19:22:54.325637 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:54.325613    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fc88b47-19fc-4246-9b30-5cde5024fc76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h\" (UID: \"4fc88b47-19fc-4246-9b30-5cde5024fc76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h"
Jul 17 19:22:54.429746 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:54.429674    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h"
Jul 17 19:22:54.570620 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:54.570590    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h"]
Jul 17 19:22:54.571271 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:22:54.571242    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc88b47_19fc_4246_9b30_5cde5024fc76.slice/crio-59983fa3d16df64b20f0b7e1b491431c66947128eed3ba4a6873b62add671cb5 WatchSource:0}: Error finding container 59983fa3d16df64b20f0b7e1b491431c66947128eed3ba4a6873b62add671cb5: Status 404 returned error can't find the container with id 59983fa3d16df64b20f0b7e1b491431c66947128eed3ba4a6873b62add671cb5
Jul 17 19:22:55.370545 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:55.370506    2559 generic.go:356] "Generic (PLEG): container finished" podID="4fc88b47-19fc-4246-9b30-5cde5024fc76" containerID="a5db604019da0e42e790794f0d300c235c0a786da8055e3b3fcbb900650c0a71" exitCode=6
Jul 17 19:22:55.370955 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:55.370598    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h" event={"ID":"4fc88b47-19fc-4246-9b30-5cde5024fc76","Type":"ContainerDied","Data":"a5db604019da0e42e790794f0d300c235c0a786da8055e3b3fcbb900650c0a71"}
Jul 17 19:22:55.370955 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:55.370688    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h" event={"ID":"4fc88b47-19fc-4246-9b30-5cde5024fc76","Type":"ContainerStarted","Data":"59983fa3d16df64b20f0b7e1b491431c66947128eed3ba4a6873b62add671cb5"}
Jul 17 19:22:55.370955 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:55.370785    2559 scope.go:117] "RemoveContainer" containerID="a5db604019da0e42e790794f0d300c235c0a786da8055e3b3fcbb900650c0a71"
Jul 17 19:22:56.377992 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:56.377958    2559 generic.go:356] "Generic (PLEG): container finished" podID="4fc88b47-19fc-4246-9b30-5cde5024fc76" containerID="48f86b3f9fb3609ecb4265ad1e498543f9113404b01a42261597f4f871986316" exitCode=6
Jul 17 19:22:56.378458 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:56.378019    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h" event={"ID":"4fc88b47-19fc-4246-9b30-5cde5024fc76","Type":"ContainerDied","Data":"48f86b3f9fb3609ecb4265ad1e498543f9113404b01a42261597f4f871986316"}
Jul 17 19:22:56.378458 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:56.378068    2559 scope.go:117] "RemoveContainer" containerID="a5db604019da0e42e790794f0d300c235c0a786da8055e3b3fcbb900650c0a71"
Jul 17 19:22:56.378458 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:56.378226    2559 scope.go:117] "RemoveContainer" containerID="48f86b3f9fb3609ecb4265ad1e498543f9113404b01a42261597f4f871986316"
Jul 17 19:22:56.378610 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:22:56.378521    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h_odh-ai-gateway-infra(4fc88b47-19fc-4246-9b30-5cde5024fc76)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h" podUID="4fc88b47-19fc-4246-9b30-5cde5024fc76"
Jul 17 19:22:57.385891 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:22:57.385867    2559 scope.go:117] "RemoveContainer" containerID="48f86b3f9fb3609ecb4265ad1e498543f9113404b01a42261597f4f871986316"
Jul 17 19:22:57.386337 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:22:57.386055    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h_odh-ai-gateway-infra(4fc88b47-19fc-4246-9b30-5cde5024fc76)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h" podUID="4fc88b47-19fc-4246-9b30-5cde5024fc76"
Jul 17 19:23:03.120298 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:03.120268    2559 scope.go:117] "RemoveContainer" containerID="72bcca5f6edea22dd7bc8318848492ff223d62c9a88099a01465c0e562bebd82"
Jul 17 19:23:04.425744 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:04.425713    2559 generic.go:356] "Generic (PLEG): container finished" podID="467e2d3b-ee06-4089-9aa7-0602eb480cfc" containerID="eb7ee3a88a17bd3350a050f1858a03ce14cb604a6a93e30468a8e2df03000bca" exitCode=6
Jul 17 19:23:04.426175 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:04.425791    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b" event={"ID":"467e2d3b-ee06-4089-9aa7-0602eb480cfc","Type":"ContainerDied","Data":"eb7ee3a88a17bd3350a050f1858a03ce14cb604a6a93e30468a8e2df03000bca"}
Jul 17 19:23:04.426175 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:04.425854    2559 scope.go:117] "RemoveContainer" containerID="72bcca5f6edea22dd7bc8318848492ff223d62c9a88099a01465c0e562bebd82"
Jul 17 19:23:04.426175 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:04.426114    2559 scope.go:117] "RemoveContainer" containerID="eb7ee3a88a17bd3350a050f1858a03ce14cb604a6a93e30468a8e2df03000bca"
Jul 17 19:23:04.426356 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:23:04.426336    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b_odh-ai-gateway-infra(467e2d3b-ee06-4089-9aa7-0602eb480cfc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b" podUID="467e2d3b-ee06-4089-9aa7-0602eb480cfc"
Jul 17 19:23:05.466471 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:05.466435    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b"]
Jul 17 19:23:05.605878 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:05.605856    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b"
Jul 17 19:23:05.729342 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:05.729225    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/467e2d3b-ee06-4089-9aa7-0602eb480cfc-maas-api-service-ca\") pod \"467e2d3b-ee06-4089-9aa7-0602eb480cfc\" (UID: \"467e2d3b-ee06-4089-9aa7-0602eb480cfc\") "
Jul 17 19:23:05.729575 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:05.729554    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467e2d3b-ee06-4089-9aa7-0602eb480cfc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "467e2d3b-ee06-4089-9aa7-0602eb480cfc" (UID: "467e2d3b-ee06-4089-9aa7-0602eb480cfc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:23:05.830874 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:05.830841    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/467e2d3b-ee06-4089-9aa7-0602eb480cfc-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:23:06.439765 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:06.439688    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b"
Jul 17 19:23:06.439765 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:06.439724    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b" event={"ID":"467e2d3b-ee06-4089-9aa7-0602eb480cfc","Type":"ContainerDied","Data":"8272f2bf4d42fc14e61cc147a28e3e2f9150fa10b5b856f9e702688938ee957e"}
Jul 17 19:23:06.439971 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:06.439784    2559 scope.go:117] "RemoveContainer" containerID="eb7ee3a88a17bd3350a050f1858a03ce14cb604a6a93e30468a8e2df03000bca"
Jul 17 19:23:06.461902 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:06.461874    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b"]
Jul 17 19:23:06.471786 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:06.471763    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-qvd6b"]
Jul 17 19:23:08.125479 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:08.125445    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="467e2d3b-ee06-4089-9aa7-0602eb480cfc" path="/var/lib/kubelet/pods/467e2d3b-ee06-4089-9aa7-0602eb480cfc/volumes"
Jul 17 19:23:10.390250 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:10.390226    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 19:23:10.397558 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:10.397536    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 19:23:10.400404 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:10.400385    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 19:23:10.407738 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:10.407717    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 19:23:11.120034 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:11.120008    2559 scope.go:117] "RemoveContainer" containerID="48f86b3f9fb3609ecb4265ad1e498543f9113404b01a42261597f4f871986316"
Jul 17 19:23:11.475813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:11.475779    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h" event={"ID":"4fc88b47-19fc-4246-9b30-5cde5024fc76","Type":"ContainerStarted","Data":"992b81953abca71ff334e30949c575a2d76abe710554ebe095f7f2d7d743aba6"}
Jul 17 19:23:11.493627 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:11.493573    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h" podStartSLOduration=17.493557798 podStartE2EDuration="17.493557798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:23:11.490856083 +0000 UTC m=+3001.932364771" watchObservedRunningTime="2026-07-17 19:23:11.493557798 +0000 UTC m=+3001.935066486"
Jul 17 19:23:12.484187 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:12.484141    2559 generic.go:356] "Generic (PLEG): container finished" podID="4fc88b47-19fc-4246-9b30-5cde5024fc76" containerID="992b81953abca71ff334e30949c575a2d76abe710554ebe095f7f2d7d743aba6" exitCode=6
Jul 17 19:23:12.484670 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:12.484219    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h" event={"ID":"4fc88b47-19fc-4246-9b30-5cde5024fc76","Type":"ContainerDied","Data":"992b81953abca71ff334e30949c575a2d76abe710554ebe095f7f2d7d743aba6"}
Jul 17 19:23:12.484670 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:12.484282    2559 scope.go:117] "RemoveContainer" containerID="48f86b3f9fb3609ecb4265ad1e498543f9113404b01a42261597f4f871986316"
Jul 17 19:23:12.484670 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:12.484536    2559 scope.go:117] "RemoveContainer" containerID="992b81953abca71ff334e30949c575a2d76abe710554ebe095f7f2d7d743aba6"
Jul 17 19:23:12.484826 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:23:12.484805    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h_odh-ai-gateway-infra(4fc88b47-19fc-4246-9b30-5cde5024fc76)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h" podUID="4fc88b47-19fc-4246-9b30-5cde5024fc76"
Jul 17 19:23:12.501465 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:12.501440    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h"]
Jul 17 19:23:13.629154 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:13.629120    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h"
Jul 17 19:23:13.805225 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:13.805113    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fc88b47-19fc-4246-9b30-5cde5024fc76-maas-api-service-ca\") pod \"4fc88b47-19fc-4246-9b30-5cde5024fc76\" (UID: \"4fc88b47-19fc-4246-9b30-5cde5024fc76\") "
Jul 17 19:23:13.805521 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:13.805498    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc88b47-19fc-4246-9b30-5cde5024fc76-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4fc88b47-19fc-4246-9b30-5cde5024fc76" (UID: "4fc88b47-19fc-4246-9b30-5cde5024fc76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:23:13.906618 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:13.906580    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fc88b47-19fc-4246-9b30-5cde5024fc76-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:23:14.499047 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:14.498961    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h"
Jul 17 19:23:14.499047 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:14.499006    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h" event={"ID":"4fc88b47-19fc-4246-9b30-5cde5024fc76","Type":"ContainerDied","Data":"59983fa3d16df64b20f0b7e1b491431c66947128eed3ba4a6873b62add671cb5"}
Jul 17 19:23:14.499284 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:14.499072    2559 scope.go:117] "RemoveContainer" containerID="992b81953abca71ff334e30949c575a2d76abe710554ebe095f7f2d7d743aba6"
Jul 17 19:23:14.515812 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:14.515784    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h"]
Jul 17 19:23:14.521401 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:14.521378    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-vrf6h"]
Jul 17 19:23:16.126412 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:16.126377    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fc88b47-19fc-4246-9b30-5cde5024fc76" path="/var/lib/kubelet/pods/4fc88b47-19fc-4246-9b30-5cde5024fc76/volumes"
Jul 17 19:23:23.004870 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.004832    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89"]
Jul 17 19:23:23.005400 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.005382    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="467e2d3b-ee06-4089-9aa7-0602eb480cfc" containerName="revoke-keys"
Jul 17 19:23:23.005457 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.005401    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="467e2d3b-ee06-4089-9aa7-0602eb480cfc" containerName="revoke-keys"
Jul 17 19:23:23.005457 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.005422    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc88b47-19fc-4246-9b30-5cde5024fc76" containerName="revoke-keys"
Jul 17 19:23:23.005457 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.005427    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc88b47-19fc-4246-9b30-5cde5024fc76" containerName="revoke-keys"
Jul 17 19:23:23.005457 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.005446    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="467e2d3b-ee06-4089-9aa7-0602eb480cfc" containerName="revoke-keys"
Jul 17 19:23:23.005457 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.005452    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="467e2d3b-ee06-4089-9aa7-0602eb480cfc" containerName="revoke-keys"
Jul 17 19:23:23.005457 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.005458    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc88b47-19fc-4246-9b30-5cde5024fc76" containerName="revoke-keys"
Jul 17 19:23:23.005644 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.005463    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc88b47-19fc-4246-9b30-5cde5024fc76" containerName="revoke-keys"
Jul 17 19:23:23.005644 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.005516    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="467e2d3b-ee06-4089-9aa7-0602eb480cfc" containerName="revoke-keys"
Jul 17 19:23:23.005644 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.005523    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc88b47-19fc-4246-9b30-5cde5024fc76" containerName="revoke-keys"
Jul 17 19:23:23.005644 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.005529    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="467e2d3b-ee06-4089-9aa7-0602eb480cfc" containerName="revoke-keys"
Jul 17 19:23:23.005644 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.005537    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc88b47-19fc-4246-9b30-5cde5024fc76" containerName="revoke-keys"
Jul 17 19:23:23.008496 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.008482    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89"
Jul 17 19:23:23.010871 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.010845    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:23:23.015582 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.015558    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89"]
Jul 17 19:23:23.088139 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.088101    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/079fd617-70d8-4e24-b40b-4f1ae27ff0c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89\" (UID: \"079fd617-70d8-4e24-b40b-4f1ae27ff0c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89"
Jul 17 19:23:23.189181 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.189140    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/079fd617-70d8-4e24-b40b-4f1ae27ff0c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89\" (UID: \"079fd617-70d8-4e24-b40b-4f1ae27ff0c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89"
Jul 17 19:23:23.189778 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.189758    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/079fd617-70d8-4e24-b40b-4f1ae27ff0c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89\" (UID: \"079fd617-70d8-4e24-b40b-4f1ae27ff0c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89"
Jul 17 19:23:23.319777 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.319747    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89"
Jul 17 19:23:23.456724 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.456697    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89"]
Jul 17 19:23:23.458016 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:23:23.457977    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079fd617_70d8_4e24_b40b_4f1ae27ff0c5.slice/crio-e24ca7dfc01226a5bf9636f454e3d9015a68ddc50d3b702b15a1b720998df780 WatchSource:0}: Error finding container e24ca7dfc01226a5bf9636f454e3d9015a68ddc50d3b702b15a1b720998df780: Status 404 returned error can't find the container with id e24ca7dfc01226a5bf9636f454e3d9015a68ddc50d3b702b15a1b720998df780
Jul 17 19:23:23.559378 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:23.559337    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89" event={"ID":"079fd617-70d8-4e24-b40b-4f1ae27ff0c5","Type":"ContainerStarted","Data":"e24ca7dfc01226a5bf9636f454e3d9015a68ddc50d3b702b15a1b720998df780"}
Jul 17 19:23:24.389515 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.389477    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"]
Jul 17 19:23:24.389903 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.389875    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc88b47-19fc-4246-9b30-5cde5024fc76" containerName="revoke-keys"
Jul 17 19:23:24.389903 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.389887    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc88b47-19fc-4246-9b30-5cde5024fc76" containerName="revoke-keys"
Jul 17 19:23:24.389996 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.389909    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="467e2d3b-ee06-4089-9aa7-0602eb480cfc" containerName="revoke-keys"
Jul 17 19:23:24.389996 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.389915    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="467e2d3b-ee06-4089-9aa7-0602eb480cfc" containerName="revoke-keys"
Jul 17 19:23:24.389996 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.389980    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc88b47-19fc-4246-9b30-5cde5024fc76" containerName="revoke-keys"
Jul 17 19:23:24.389996 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.389992    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="467e2d3b-ee06-4089-9aa7-0602eb480cfc" containerName="revoke-keys"
Jul 17 19:23:24.392902 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.392882    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:24.395356 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.395333    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4aee7c\"/\"default-dockercfg-fh9s4\""
Jul 17 19:23:24.395356 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.395354    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4aee7c\"/\"openshift-service-ca.crt\""
Jul 17 19:23:24.396302 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.396282    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4aee7c\"/\"kube-root-ca.crt\""
Jul 17 19:23:24.396492 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.396472    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4aee7c\"/\"test-model-4aee7c-kserve-self-signed-certs\""
Jul 17 19:23:24.401559 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.401540    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"]
Jul 17 19:23:24.499965 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.499936    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-home\") pod \"test-model-4aee7c-kserve-79b74ff7f9-569km\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") " pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:24.500146 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.499989    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-dshm\") pod \"test-model-4aee7c-kserve-79b74ff7f9-569km\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") " pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:24.500146 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.500076    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-model-cache\") pod \"test-model-4aee7c-kserve-79b74ff7f9-569km\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") " pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:24.500146 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.500109    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-tmp-dir\") pod \"test-model-4aee7c-kserve-79b74ff7f9-569km\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") " pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:24.500360 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.500180    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/794ccbd1-2557-4fc1-ba9d-a6783232f01a-tls-certs\") pod \"test-model-4aee7c-kserve-79b74ff7f9-569km\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") " pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:24.500360 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.500221    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d4tr\" (UniqueName: \"kubernetes.io/projected/794ccbd1-2557-4fc1-ba9d-a6783232f01a-kube-api-access-9d4tr\") pod \"test-model-4aee7c-kserve-79b74ff7f9-569km\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") " pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:24.566608 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.566575    2559 generic.go:356] "Generic (PLEG): container finished" podID="079fd617-70d8-4e24-b40b-4f1ae27ff0c5" containerID="b02a9e99fd87e7880774718f94003955d36df18520f0339c0c366cb1472e55e0" exitCode=6
Jul 17 19:23:24.566762 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.566644    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89" event={"ID":"079fd617-70d8-4e24-b40b-4f1ae27ff0c5","Type":"ContainerDied","Data":"b02a9e99fd87e7880774718f94003955d36df18520f0339c0c366cb1472e55e0"}
Jul 17 19:23:24.566827 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.566813    2559 scope.go:117] "RemoveContainer" containerID="b02a9e99fd87e7880774718f94003955d36df18520f0339c0c366cb1472e55e0"
Jul 17 19:23:24.600724 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.600696    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-dshm\") pod \"test-model-4aee7c-kserve-79b74ff7f9-569km\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") " pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:24.600824 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.600749    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-model-cache\") pod \"test-model-4aee7c-kserve-79b74ff7f9-569km\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") " pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:24.600824 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.600774    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-tmp-dir\") pod \"test-model-4aee7c-kserve-79b74ff7f9-569km\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") " pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:24.600824 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.600807    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/794ccbd1-2557-4fc1-ba9d-a6783232f01a-tls-certs\") pod \"test-model-4aee7c-kserve-79b74ff7f9-569km\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") " pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:24.600962 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.600941    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d4tr\" (UniqueName: \"kubernetes.io/projected/794ccbd1-2557-4fc1-ba9d-a6783232f01a-kube-api-access-9d4tr\") pod \"test-model-4aee7c-kserve-79b74ff7f9-569km\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") " pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:24.601062 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.601043    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-home\") pod \"test-model-4aee7c-kserve-79b74ff7f9-569km\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") " pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:24.601148 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.601106    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-model-cache\") pod \"test-model-4aee7c-kserve-79b74ff7f9-569km\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") " pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:24.601209 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.601168    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-tmp-dir\") pod \"test-model-4aee7c-kserve-79b74ff7f9-569km\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") " pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:24.601442 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.601423    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-home\") pod \"test-model-4aee7c-kserve-79b74ff7f9-569km\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") " pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:24.603025 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.603008    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-dshm\") pod \"test-model-4aee7c-kserve-79b74ff7f9-569km\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") " pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:24.603281 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.603264    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/794ccbd1-2557-4fc1-ba9d-a6783232f01a-tls-certs\") pod \"test-model-4aee7c-kserve-79b74ff7f9-569km\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") " pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:24.609418 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.609399    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d4tr\" (UniqueName: \"kubernetes.io/projected/794ccbd1-2557-4fc1-ba9d-a6783232f01a-kube-api-access-9d4tr\") pod \"test-model-4aee7c-kserve-79b74ff7f9-569km\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") " pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:24.703617 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.703541    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:24.847289 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:24.847264    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"]
Jul 17 19:23:24.848802 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:23:24.848779    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod794ccbd1_2557_4fc1_ba9d_a6783232f01a.slice/crio-325470eb04f75c49c9914c2dc0dceb746aee78b5b6a653450393a727491a1de4 WatchSource:0}: Error finding container 325470eb04f75c49c9914c2dc0dceb746aee78b5b6a653450393a727491a1de4: Status 404 returned error can't find the container with id 325470eb04f75c49c9914c2dc0dceb746aee78b5b6a653450393a727491a1de4
Jul 17 19:23:25.573792 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:25.573752    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km" event={"ID":"794ccbd1-2557-4fc1-ba9d-a6783232f01a","Type":"ContainerStarted","Data":"038c8575ff4c08b1387286e23a81f7a935a3d7789ca912b378a0888c7ef5b062"}
Jul 17 19:23:25.574250 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:25.573799    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km" event={"ID":"794ccbd1-2557-4fc1-ba9d-a6783232f01a","Type":"ContainerStarted","Data":"325470eb04f75c49c9914c2dc0dceb746aee78b5b6a653450393a727491a1de4"}
Jul 17 19:23:25.575557 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:25.575533    2559 generic.go:356] "Generic (PLEG): container finished" podID="079fd617-70d8-4e24-b40b-4f1ae27ff0c5" containerID="f4cd0c02bd65954bf04b1ebddc6af8e4009ce6918095b409edd2d20a0ebce266" exitCode=6
Jul 17 19:23:25.575644 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:25.575618    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89" event={"ID":"079fd617-70d8-4e24-b40b-4f1ae27ff0c5","Type":"ContainerDied","Data":"f4cd0c02bd65954bf04b1ebddc6af8e4009ce6918095b409edd2d20a0ebce266"}
Jul 17 19:23:25.575690 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:25.575674    2559 scope.go:117] "RemoveContainer" containerID="b02a9e99fd87e7880774718f94003955d36df18520f0339c0c366cb1472e55e0"
Jul 17 19:23:25.575760 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:25.575745    2559 scope.go:117] "RemoveContainer" containerID="f4cd0c02bd65954bf04b1ebddc6af8e4009ce6918095b409edd2d20a0ebce266"
Jul 17 19:23:25.575986 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:23:25.575967    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89_odh-ai-gateway-infra(079fd617-70d8-4e24-b40b-4f1ae27ff0c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89" podUID="079fd617-70d8-4e24-b40b-4f1ae27ff0c5"
Jul 17 19:23:25.591553 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:25.591512    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km" podStartSLOduration=1.5914966910000001 podStartE2EDuration="1.591496691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:23:25.59052093 +0000 UTC m=+3016.032029619" watchObservedRunningTime="2026-07-17 19:23:25.591496691 +0000 UTC m=+3016.033005379"
Jul 17 19:23:26.583781 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:26.583756    2559 scope.go:117] "RemoveContainer" containerID="f4cd0c02bd65954bf04b1ebddc6af8e4009ce6918095b409edd2d20a0ebce266"
Jul 17 19:23:26.584176 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:23:26.583933    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89_odh-ai-gateway-infra(079fd617-70d8-4e24-b40b-4f1ae27ff0c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89" podUID="079fd617-70d8-4e24-b40b-4f1ae27ff0c5"
Jul 17 19:23:34.704574 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:34.704538    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:34.704972 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:34.704596    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:34.716855 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:34.716823    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:35.643603 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:35.643568    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:23:38.119911 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:38.119876    2559 scope.go:117] "RemoveContainer" containerID="f4cd0c02bd65954bf04b1ebddc6af8e4009ce6918095b409edd2d20a0ebce266"
Jul 17 19:23:38.658027 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:38.657991    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89" event={"ID":"079fd617-70d8-4e24-b40b-4f1ae27ff0c5","Type":"ContainerStarted","Data":"7d38e5a3c94593d6f65ef7b0f2373d0cec09167417ed80b2b2d8b9ceb4979ee7"}
Jul 17 19:23:38.676195 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:38.676146    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89" podStartSLOduration=16.676132455 podStartE2EDuration="16.676132455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:23:38.675005893 +0000 UTC m=+3029.116514585" watchObservedRunningTime="2026-07-17 19:23:38.676132455 +0000 UTC m=+3029.117641144"
Jul 17 19:23:39.665861 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:39.665832    2559 generic.go:356] "Generic (PLEG): container finished" podID="079fd617-70d8-4e24-b40b-4f1ae27ff0c5" containerID="7d38e5a3c94593d6f65ef7b0f2373d0cec09167417ed80b2b2d8b9ceb4979ee7" exitCode=6
Jul 17 19:23:39.666299 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:39.665875    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89" event={"ID":"079fd617-70d8-4e24-b40b-4f1ae27ff0c5","Type":"ContainerDied","Data":"7d38e5a3c94593d6f65ef7b0f2373d0cec09167417ed80b2b2d8b9ceb4979ee7"}
Jul 17 19:23:39.666299 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:39.665921    2559 scope.go:117] "RemoveContainer" containerID="f4cd0c02bd65954bf04b1ebddc6af8e4009ce6918095b409edd2d20a0ebce266"
Jul 17 19:23:39.666299 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:39.666205    2559 scope.go:117] "RemoveContainer" containerID="7d38e5a3c94593d6f65ef7b0f2373d0cec09167417ed80b2b2d8b9ceb4979ee7"
Jul 17 19:23:39.666485 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:23:39.666459    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89_odh-ai-gateway-infra(079fd617-70d8-4e24-b40b-4f1ae27ff0c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89" podUID="079fd617-70d8-4e24-b40b-4f1ae27ff0c5"
Jul 17 19:23:39.692692 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:39.692667    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89"]
Jul 17 19:23:40.813724 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:40.813698    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89"
Jul 17 19:23:40.946660 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:40.946579    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/079fd617-70d8-4e24-b40b-4f1ae27ff0c5-maas-api-service-ca\") pod \"079fd617-70d8-4e24-b40b-4f1ae27ff0c5\" (UID: \"079fd617-70d8-4e24-b40b-4f1ae27ff0c5\") "
Jul 17 19:23:40.946922 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:40.946901    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/079fd617-70d8-4e24-b40b-4f1ae27ff0c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "079fd617-70d8-4e24-b40b-4f1ae27ff0c5" (UID: "079fd617-70d8-4e24-b40b-4f1ae27ff0c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:23:41.048122 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.048083    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/079fd617-70d8-4e24-b40b-4f1ae27ff0c5-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:23:41.281641 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.281561    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf"]
Jul 17 19:23:41.282053 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.282028    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="079fd617-70d8-4e24-b40b-4f1ae27ff0c5" containerName="revoke-keys"
Jul 17 19:23:41.282053 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.282053    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="079fd617-70d8-4e24-b40b-4f1ae27ff0c5" containerName="revoke-keys"
Jul 17 19:23:41.282219 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.282067    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="079fd617-70d8-4e24-b40b-4f1ae27ff0c5" containerName="revoke-keys"
Jul 17 19:23:41.282219 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.282076    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="079fd617-70d8-4e24-b40b-4f1ae27ff0c5" containerName="revoke-keys"
Jul 17 19:23:41.282219 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.282094    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="079fd617-70d8-4e24-b40b-4f1ae27ff0c5" containerName="revoke-keys"
Jul 17 19:23:41.282219 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.282102    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="079fd617-70d8-4e24-b40b-4f1ae27ff0c5" containerName="revoke-keys"
Jul 17 19:23:41.282366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.282258    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="079fd617-70d8-4e24-b40b-4f1ae27ff0c5" containerName="revoke-keys"
Jul 17 19:23:41.282366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.282271    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="079fd617-70d8-4e24-b40b-4f1ae27ff0c5" containerName="revoke-keys"
Jul 17 19:23:41.282366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.282284    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="079fd617-70d8-4e24-b40b-4f1ae27ff0c5" containerName="revoke-keys"
Jul 17 19:23:41.285352 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.285333    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf"
Jul 17 19:23:41.293878 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.293522    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf"]
Jul 17 19:23:41.351106 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.351075    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/febb45f9-0b54-4579-ac94-759fbd997aa1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf\" (UID: \"febb45f9-0b54-4579-ac94-759fbd997aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf"
Jul 17 19:23:41.452115 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.452079    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/febb45f9-0b54-4579-ac94-759fbd997aa1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf\" (UID: \"febb45f9-0b54-4579-ac94-759fbd997aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf"
Jul 17 19:23:41.452730 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.452711    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/febb45f9-0b54-4579-ac94-759fbd997aa1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf\" (UID: \"febb45f9-0b54-4579-ac94-759fbd997aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf"
Jul 17 19:23:41.598087 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.598060    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf"
Jul 17 19:23:41.682829 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.682799    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89" event={"ID":"079fd617-70d8-4e24-b40b-4f1ae27ff0c5","Type":"ContainerDied","Data":"e24ca7dfc01226a5bf9636f454e3d9015a68ddc50d3b702b15a1b720998df780"}
Jul 17 19:23:41.682829 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.682820    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89"
Jul 17 19:23:41.683008 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.682868    2559 scope.go:117] "RemoveContainer" containerID="7d38e5a3c94593d6f65ef7b0f2373d0cec09167417ed80b2b2d8b9ceb4979ee7"
Jul 17 19:23:41.712291 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.712258    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89"]
Jul 17 19:23:41.714942 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.714918    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-q4h89"]
Jul 17 19:23:41.735753 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:41.735729    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf"]
Jul 17 19:23:41.736754 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:23:41.736728    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfebb45f9_0b54_4579_ac94_759fbd997aa1.slice/crio-42038527af9e82915eceb767294f9617fcfaf1dd5ecc23f03541745e61c162fe WatchSource:0}: Error finding container 42038527af9e82915eceb767294f9617fcfaf1dd5ecc23f03541745e61c162fe: Status 404 returned error can't find the container with id 42038527af9e82915eceb767294f9617fcfaf1dd5ecc23f03541745e61c162fe
Jul 17 19:23:42.125667 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:42.125635    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="079fd617-70d8-4e24-b40b-4f1ae27ff0c5" path="/var/lib/kubelet/pods/079fd617-70d8-4e24-b40b-4f1ae27ff0c5/volumes"
Jul 17 19:23:42.697053 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:42.696959    2559 generic.go:356] "Generic (PLEG): container finished" podID="febb45f9-0b54-4579-ac94-759fbd997aa1" containerID="bb7e983a72795535b135009e5590f7871838d0e386664a000468e25123054e93" exitCode=6
Jul 17 19:23:42.697238 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:42.697048    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf" event={"ID":"febb45f9-0b54-4579-ac94-759fbd997aa1","Type":"ContainerDied","Data":"bb7e983a72795535b135009e5590f7871838d0e386664a000468e25123054e93"}
Jul 17 19:23:42.697238 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:42.697151    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf" event={"ID":"febb45f9-0b54-4579-ac94-759fbd997aa1","Type":"ContainerStarted","Data":"42038527af9e82915eceb767294f9617fcfaf1dd5ecc23f03541745e61c162fe"}
Jul 17 19:23:42.697238 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:42.697216    2559 scope.go:117] "RemoveContainer" containerID="bb7e983a72795535b135009e5590f7871838d0e386664a000468e25123054e93"
Jul 17 19:23:43.705522 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:43.705436    2559 generic.go:356] "Generic (PLEG): container finished" podID="febb45f9-0b54-4579-ac94-759fbd997aa1" containerID="db71541220d49dffcbb923a792aa5547e3fecc67535a2f472d6486482dc73e1b" exitCode=6
Jul 17 19:23:43.705522 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:43.705485    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf" event={"ID":"febb45f9-0b54-4579-ac94-759fbd997aa1","Type":"ContainerDied","Data":"db71541220d49dffcbb923a792aa5547e3fecc67535a2f472d6486482dc73e1b"}
Jul 17 19:23:43.705990 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:43.705558    2559 scope.go:117] "RemoveContainer" containerID="bb7e983a72795535b135009e5590f7871838d0e386664a000468e25123054e93"
Jul 17 19:23:43.705990 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:43.705760    2559 scope.go:117] "RemoveContainer" containerID="db71541220d49dffcbb923a792aa5547e3fecc67535a2f472d6486482dc73e1b"
Jul 17 19:23:43.706082 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:23:43.706001    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf_odh-ai-gateway-infra(febb45f9-0b54-4579-ac94-759fbd997aa1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf" podUID="febb45f9-0b54-4579-ac94-759fbd997aa1"
Jul 17 19:23:44.713182 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:44.713152    2559 scope.go:117] "RemoveContainer" containerID="db71541220d49dffcbb923a792aa5547e3fecc67535a2f472d6486482dc73e1b"
Jul 17 19:23:44.713593 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:23:44.713419    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf_odh-ai-gateway-infra(febb45f9-0b54-4579-ac94-759fbd997aa1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf" podUID="febb45f9-0b54-4579-ac94-759fbd997aa1"
Jul 17 19:23:45.503347 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.503293    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"]
Jul 17 19:23:45.508177 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.508162    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:45.511604 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.511566    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-15f29d\"/\"kube-root-ca.crt\""
Jul 17 19:23:45.511604 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.511567    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-15f29d\"/\"test-model-15f29d-kserve-self-signed-certs\""
Jul 17 19:23:45.511854 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.511613    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-15f29d\"/\"openshift-service-ca.crt\""
Jul 17 19:23:45.511854 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.511567    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-15f29d\"/\"default-dockercfg-nprrb\""
Jul 17 19:23:45.517745 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.517724    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"]
Jul 17 19:23:45.593910 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.593880    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-tmp-dir\") pod \"test-model-15f29d-kserve-786977d759-r9g28\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") " pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:45.594102 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.593918    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5nlb\" (UniqueName: \"kubernetes.io/projected/01d0d0c4-bade-428d-a697-a71aa6875663-kube-api-access-z5nlb\") pod \"test-model-15f29d-kserve-786977d759-r9g28\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") " pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:45.594102 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.593976    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-dshm\") pod \"test-model-15f29d-kserve-786977d759-r9g28\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") " pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:45.594102 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.593989    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01d0d0c4-bade-428d-a697-a71aa6875663-tls-certs\") pod \"test-model-15f29d-kserve-786977d759-r9g28\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") " pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:45.594102 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.594013    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-home\") pod \"test-model-15f29d-kserve-786977d759-r9g28\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") " pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:45.594102 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.594057    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-model-cache\") pod \"test-model-15f29d-kserve-786977d759-r9g28\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") " pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:45.695050 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.695022    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-dshm\") pod \"test-model-15f29d-kserve-786977d759-r9g28\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") " pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:45.695050 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.695052    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01d0d0c4-bade-428d-a697-a71aa6875663-tls-certs\") pod \"test-model-15f29d-kserve-786977d759-r9g28\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") " pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:45.695345 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.695081    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-home\") pod \"test-model-15f29d-kserve-786977d759-r9g28\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") " pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:45.695345 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.695107    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-model-cache\") pod \"test-model-15f29d-kserve-786977d759-r9g28\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") " pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:45.695345 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.695181    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-tmp-dir\") pod \"test-model-15f29d-kserve-786977d759-r9g28\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") " pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:45.695523 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.695338    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5nlb\" (UniqueName: \"kubernetes.io/projected/01d0d0c4-bade-428d-a697-a71aa6875663-kube-api-access-z5nlb\") pod \"test-model-15f29d-kserve-786977d759-r9g28\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") " pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:45.695578 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.695542    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-home\") pod \"test-model-15f29d-kserve-786977d759-r9g28\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") " pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:45.695578 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.695560    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-tmp-dir\") pod \"test-model-15f29d-kserve-786977d759-r9g28\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") " pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:45.695654 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.695620    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-model-cache\") pod \"test-model-15f29d-kserve-786977d759-r9g28\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") " pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:45.697538 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.697521    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-dshm\") pod \"test-model-15f29d-kserve-786977d759-r9g28\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") " pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:45.697926 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.697904    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01d0d0c4-bade-428d-a697-a71aa6875663-tls-certs\") pod \"test-model-15f29d-kserve-786977d759-r9g28\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") " pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:45.704037 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.704020    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5nlb\" (UniqueName: \"kubernetes.io/projected/01d0d0c4-bade-428d-a697-a71aa6875663-kube-api-access-z5nlb\") pod \"test-model-15f29d-kserve-786977d759-r9g28\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") " pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:45.818706 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.818678    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:45.961064 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:45.961039    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"]
Jul 17 19:23:45.962583 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:23:45.962550    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01d0d0c4_bade_428d_a697_a71aa6875663.slice/crio-9785e75d10d8df7369c6ee796c7f42ca1524415e5ea8d540ea70751710b96e49 WatchSource:0}: Error finding container 9785e75d10d8df7369c6ee796c7f42ca1524415e5ea8d540ea70751710b96e49: Status 404 returned error can't find the container with id 9785e75d10d8df7369c6ee796c7f42ca1524415e5ea8d540ea70751710b96e49
Jul 17 19:23:46.727026 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:46.726993    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28" event={"ID":"01d0d0c4-bade-428d-a697-a71aa6875663","Type":"ContainerStarted","Data":"21061cb38ce7188530fd7f952bbb747bf8551c5ba8b7bbdd5bab6c880662f6ce"}
Jul 17 19:23:46.727227 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:46.727034    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28" event={"ID":"01d0d0c4-bade-428d-a697-a71aa6875663","Type":"ContainerStarted","Data":"9785e75d10d8df7369c6ee796c7f42ca1524415e5ea8d540ea70751710b96e49"}
Jul 17 19:23:46.748344 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:46.748273    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28" podStartSLOduration=1.748254372 podStartE2EDuration="1.748254372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:23:46.745370327 +0000 UTC m=+3037.186879015" watchObservedRunningTime="2026-07-17 19:23:46.748254372 +0000 UTC m=+3037.189763062"
Jul 17 19:23:54.164988 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:54.164954    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7"]
Jul 17 19:23:54.168484 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:54.168469    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7"
Jul 17 19:23:54.175019 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:54.174991    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7"]
Jul 17 19:23:54.280502 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:54.280467    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df312190-1533-4d48-b790-701bba98cae8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7\" (UID: \"df312190-1533-4d48-b790-701bba98cae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7"
Jul 17 19:23:54.381987 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:54.381951    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df312190-1533-4d48-b790-701bba98cae8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7\" (UID: \"df312190-1533-4d48-b790-701bba98cae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7"
Jul 17 19:23:54.382599 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:54.382578    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df312190-1533-4d48-b790-701bba98cae8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7\" (UID: \"df312190-1533-4d48-b790-701bba98cae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7"
Jul 17 19:23:54.480947 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:54.480872    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7"
Jul 17 19:23:54.618036 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:54.618011    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7"]
Jul 17 19:23:54.619049 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:23:54.619002    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf312190_1533_4d48_b790_701bba98cae8.slice/crio-9c325dc83b8b1c681c394e2eaf6612a4fe05b79ea19808765ff2460c8ccf9c57 WatchSource:0}: Error finding container 9c325dc83b8b1c681c394e2eaf6612a4fe05b79ea19808765ff2460c8ccf9c57: Status 404 returned error can't find the container with id 9c325dc83b8b1c681c394e2eaf6612a4fe05b79ea19808765ff2460c8ccf9c57
Jul 17 19:23:54.775456 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:54.775420    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7" event={"ID":"df312190-1533-4d48-b790-701bba98cae8","Type":"ContainerStarted","Data":"9c325dc83b8b1c681c394e2eaf6612a4fe05b79ea19808765ff2460c8ccf9c57"}
Jul 17 19:23:55.783625 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:55.783587    2559 generic.go:356] "Generic (PLEG): container finished" podID="df312190-1533-4d48-b790-701bba98cae8" containerID="66d006f65059d28cac152ee6a043f712cdf16514e088bdaee07815e453472bee" exitCode=6
Jul 17 19:23:55.784104 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:55.783673    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7" event={"ID":"df312190-1533-4d48-b790-701bba98cae8","Type":"ContainerDied","Data":"66d006f65059d28cac152ee6a043f712cdf16514e088bdaee07815e453472bee"}
Jul 17 19:23:55.784104 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:55.783858    2559 scope.go:117] "RemoveContainer" containerID="66d006f65059d28cac152ee6a043f712cdf16514e088bdaee07815e453472bee"
Jul 17 19:23:55.819033 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:55.819009    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:55.819131 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:55.819047    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:55.832050 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:55.832022    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:56.792472 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:56.792380    2559 generic.go:356] "Generic (PLEG): container finished" podID="df312190-1533-4d48-b790-701bba98cae8" containerID="cba668f19ed6124175c167953087b70312d0beeb073d2e0f81729e4cbc52a3fd" exitCode=6
Jul 17 19:23:56.792472 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:56.792457    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7" event={"ID":"df312190-1533-4d48-b790-701bba98cae8","Type":"ContainerDied","Data":"cba668f19ed6124175c167953087b70312d0beeb073d2e0f81729e4cbc52a3fd"}
Jul 17 19:23:56.792980 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:56.792499    2559 scope.go:117] "RemoveContainer" containerID="66d006f65059d28cac152ee6a043f712cdf16514e088bdaee07815e453472bee"
Jul 17 19:23:56.792980 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:56.792713    2559 scope.go:117] "RemoveContainer" containerID="cba668f19ed6124175c167953087b70312d0beeb073d2e0f81729e4cbc52a3fd"
Jul 17 19:23:56.792980 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:23:56.792963    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7_odh-ai-gateway-infra(df312190-1533-4d48-b790-701bba98cae8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7" podUID="df312190-1533-4d48-b790-701bba98cae8"
Jul 17 19:23:56.804646 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:56.804626    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:23:57.120201 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:57.120170    2559 scope.go:117] "RemoveContainer" containerID="db71541220d49dffcbb923a792aa5547e3fecc67535a2f472d6486482dc73e1b"
Jul 17 19:23:57.800469 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:57.800444    2559 scope.go:117] "RemoveContainer" containerID="cba668f19ed6124175c167953087b70312d0beeb073d2e0f81729e4cbc52a3fd"
Jul 17 19:23:57.800869 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:23:57.800708    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7_odh-ai-gateway-infra(df312190-1533-4d48-b790-701bba98cae8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7" podUID="df312190-1533-4d48-b790-701bba98cae8"
Jul 17 19:23:57.802095 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:57.802069    2559 generic.go:356] "Generic (PLEG): container finished" podID="febb45f9-0b54-4579-ac94-759fbd997aa1" containerID="408c1fd770cd78296381efa00479fa8b30401e0094d74009b0f44ef996e69def" exitCode=6
Jul 17 19:23:57.802208 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:57.802144    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf" event={"ID":"febb45f9-0b54-4579-ac94-759fbd997aa1","Type":"ContainerDied","Data":"408c1fd770cd78296381efa00479fa8b30401e0094d74009b0f44ef996e69def"}
Jul 17 19:23:57.802208 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:57.802203    2559 scope.go:117] "RemoveContainer" containerID="db71541220d49dffcbb923a792aa5547e3fecc67535a2f472d6486482dc73e1b"
Jul 17 19:23:57.802613 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:57.802595    2559 scope.go:117] "RemoveContainer" containerID="408c1fd770cd78296381efa00479fa8b30401e0094d74009b0f44ef996e69def"
Jul 17 19:23:57.802811 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:23:57.802787    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf_odh-ai-gateway-infra(febb45f9-0b54-4579-ac94-759fbd997aa1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf" podUID="febb45f9-0b54-4579-ac94-759fbd997aa1"
Jul 17 19:23:58.847067 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:58.847033    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf"]
Jul 17 19:23:58.983678 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:58.983652    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf"
Jul 17 19:23:59.130053 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:59.129975    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/febb45f9-0b54-4579-ac94-759fbd997aa1-maas-api-service-ca\") pod \"febb45f9-0b54-4579-ac94-759fbd997aa1\" (UID: \"febb45f9-0b54-4579-ac94-759fbd997aa1\") "
Jul 17 19:23:59.130359 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:59.130338    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/febb45f9-0b54-4579-ac94-759fbd997aa1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "febb45f9-0b54-4579-ac94-759fbd997aa1" (UID: "febb45f9-0b54-4579-ac94-759fbd997aa1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:23:59.231436 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:59.231404    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/febb45f9-0b54-4579-ac94-759fbd997aa1-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:23:59.817784 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:59.817749    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf" event={"ID":"febb45f9-0b54-4579-ac94-759fbd997aa1","Type":"ContainerDied","Data":"42038527af9e82915eceb767294f9617fcfaf1dd5ecc23f03541745e61c162fe"}
Jul 17 19:23:59.817784 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:59.817781    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf"
Jul 17 19:23:59.818040 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:59.817813    2559 scope.go:117] "RemoveContainer" containerID="408c1fd770cd78296381efa00479fa8b30401e0094d74009b0f44ef996e69def"
Jul 17 19:23:59.840721 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:59.840696    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf"]
Jul 17 19:23:59.844390 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:23:59.844353    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-qnsqf"]
Jul 17 19:24:00.125974 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:00.125865    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="febb45f9-0b54-4579-ac94-759fbd997aa1" path="/var/lib/kubelet/pods/febb45f9-0b54-4579-ac94-759fbd997aa1/volumes"
Jul 17 19:24:11.120299 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:11.120268    2559 scope.go:117] "RemoveContainer" containerID="cba668f19ed6124175c167953087b70312d0beeb073d2e0f81729e4cbc52a3fd"
Jul 17 19:24:11.457831 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:11.457800    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"]
Jul 17 19:24:11.888426 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:11.888392    2559 generic.go:356] "Generic (PLEG): container finished" podID="df312190-1533-4d48-b790-701bba98cae8" containerID="32c016f6395e6b5bcbe8162c052f80efd3c0e9a75a93f2e1f6cf2010efcf3c4d" exitCode=6
Jul 17 19:24:11.888565 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:11.888462    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7" event={"ID":"df312190-1533-4d48-b790-701bba98cae8","Type":"ContainerDied","Data":"32c016f6395e6b5bcbe8162c052f80efd3c0e9a75a93f2e1f6cf2010efcf3c4d"}
Jul 17 19:24:11.888565 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:11.888526    2559 scope.go:117] "RemoveContainer" containerID="cba668f19ed6124175c167953087b70312d0beeb073d2e0f81729e4cbc52a3fd"
Jul 17 19:24:11.888837 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:11.888815    2559 scope.go:117] "RemoveContainer" containerID="32c016f6395e6b5bcbe8162c052f80efd3c0e9a75a93f2e1f6cf2010efcf3c4d"
Jul 17 19:24:11.889082 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:24:11.889064    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7_odh-ai-gateway-infra(df312190-1533-4d48-b790-701bba98cae8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7" podUID="df312190-1533-4d48-b790-701bba98cae8"
Jul 17 19:24:12.336684 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:12.336650    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"]
Jul 17 19:24:12.919052 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:12.919012    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7"]
Jul 17 19:24:13.055600 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:13.055575    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7"
Jul 17 19:24:13.162835 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:13.162802    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df312190-1533-4d48-b790-701bba98cae8-maas-api-service-ca\") pod \"df312190-1533-4d48-b790-701bba98cae8\" (UID: \"df312190-1533-4d48-b790-701bba98cae8\") "
Jul 17 19:24:13.163181 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:13.163161    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df312190-1533-4d48-b790-701bba98cae8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "df312190-1533-4d48-b790-701bba98cae8" (UID: "df312190-1533-4d48-b790-701bba98cae8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:24:13.263668 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:13.263594    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df312190-1533-4d48-b790-701bba98cae8-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:24:13.902393 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:13.902367    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7"
Jul 17 19:24:13.902825 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:13.902402    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7" event={"ID":"df312190-1533-4d48-b790-701bba98cae8","Type":"ContainerDied","Data":"9c325dc83b8b1c681c394e2eaf6612a4fe05b79ea19808765ff2460c8ccf9c57"}
Jul 17 19:24:13.902825 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:13.902467    2559 scope.go:117] "RemoveContainer" containerID="32c016f6395e6b5bcbe8162c052f80efd3c0e9a75a93f2e1f6cf2010efcf3c4d"
Jul 17 19:24:13.923717 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:13.923682    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7"]
Jul 17 19:24:13.929490 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:13.929466    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-r6qv7"]
Jul 17 19:24:14.125740 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:14.125710    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df312190-1533-4d48-b790-701bba98cae8" path="/var/lib/kubelet/pods/df312190-1533-4d48-b790-701bba98cae8/volumes"
Jul 17 19:24:26.478296 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.478257    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km" podUID="794ccbd1-2557-4fc1-ba9d-a6783232f01a" containerName="main" containerID="cri-o://038c8575ff4c08b1387286e23a81f7a935a3d7789ca912b378a0888c7ef5b062" gracePeriod=45
Jul 17 19:24:26.729986 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.729931    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:24:26.900036 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.900011    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-home\") pod \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") "
Jul 17 19:24:26.900350 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.900057    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-tmp-dir\") pod \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") "
Jul 17 19:24:26.900350 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.900138    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-dshm\") pod \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") "
Jul 17 19:24:26.900350 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.900161    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-model-cache\") pod \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") "
Jul 17 19:24:26.900350 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.900227    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/794ccbd1-2557-4fc1-ba9d-a6783232f01a-tls-certs\") pod \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") "
Jul 17 19:24:26.900350 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.900261    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d4tr\" (UniqueName: \"kubernetes.io/projected/794ccbd1-2557-4fc1-ba9d-a6783232f01a-kube-api-access-9d4tr\") pod \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\" (UID: \"794ccbd1-2557-4fc1-ba9d-a6783232f01a\") "
Jul 17 19:24:26.900350 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.900296    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-home" (OuterVolumeSpecName: "home") pod "794ccbd1-2557-4fc1-ba9d-a6783232f01a" (UID: "794ccbd1-2557-4fc1-ba9d-a6783232f01a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:24:26.900679 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.900368    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "794ccbd1-2557-4fc1-ba9d-a6783232f01a" (UID: "794ccbd1-2557-4fc1-ba9d-a6783232f01a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:24:26.900679 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.900451    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-model-cache" (OuterVolumeSpecName: "model-cache") pod "794ccbd1-2557-4fc1-ba9d-a6783232f01a" (UID: "794ccbd1-2557-4fc1-ba9d-a6783232f01a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:24:26.900679 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.900573    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-model-cache\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:24:26.900679 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.900587    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-home\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:24:26.900679 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.900599    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-tmp-dir\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:24:26.902575 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.902555    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-dshm" (OuterVolumeSpecName: "dshm") pod "794ccbd1-2557-4fc1-ba9d-a6783232f01a" (UID: "794ccbd1-2557-4fc1-ba9d-a6783232f01a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:24:26.902662 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.902592    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/794ccbd1-2557-4fc1-ba9d-a6783232f01a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "794ccbd1-2557-4fc1-ba9d-a6783232f01a" (UID: "794ccbd1-2557-4fc1-ba9d-a6783232f01a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:24:26.902710 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.902674    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794ccbd1-2557-4fc1-ba9d-a6783232f01a-kube-api-access-9d4tr" (OuterVolumeSpecName: "kube-api-access-9d4tr") pod "794ccbd1-2557-4fc1-ba9d-a6783232f01a" (UID: "794ccbd1-2557-4fc1-ba9d-a6783232f01a"). InnerVolumeSpecName "kube-api-access-9d4tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:24:26.976980 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.976949    2559 generic.go:356] "Generic (PLEG): container finished" podID="794ccbd1-2557-4fc1-ba9d-a6783232f01a" containerID="038c8575ff4c08b1387286e23a81f7a935a3d7789ca912b378a0888c7ef5b062" exitCode=0
Jul 17 19:24:26.977096 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.976993    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"
Jul 17 19:24:26.977096 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.977019    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km" event={"ID":"794ccbd1-2557-4fc1-ba9d-a6783232f01a","Type":"ContainerDied","Data":"038c8575ff4c08b1387286e23a81f7a935a3d7789ca912b378a0888c7ef5b062"}
Jul 17 19:24:26.977096 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.977051    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km" event={"ID":"794ccbd1-2557-4fc1-ba9d-a6783232f01a","Type":"ContainerDied","Data":"325470eb04f75c49c9914c2dc0dceb746aee78b5b6a653450393a727491a1de4"}
Jul 17 19:24:26.977096 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.977066    2559 scope.go:117] "RemoveContainer" containerID="038c8575ff4c08b1387286e23a81f7a935a3d7789ca912b378a0888c7ef5b062"
Jul 17 19:24:26.987276 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.987259    2559 scope.go:117] "RemoveContainer" containerID="038c8575ff4c08b1387286e23a81f7a935a3d7789ca912b378a0888c7ef5b062"
Jul 17 19:24:26.987562 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:24:26.987543    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"038c8575ff4c08b1387286e23a81f7a935a3d7789ca912b378a0888c7ef5b062\": container with ID starting with 038c8575ff4c08b1387286e23a81f7a935a3d7789ca912b378a0888c7ef5b062 not found: ID does not exist" containerID="038c8575ff4c08b1387286e23a81f7a935a3d7789ca912b378a0888c7ef5b062"
Jul 17 19:24:26.987611 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:26.987570    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038c8575ff4c08b1387286e23a81f7a935a3d7789ca912b378a0888c7ef5b062"} err="failed to get container status \"038c8575ff4c08b1387286e23a81f7a935a3d7789ca912b378a0888c7ef5b062\": rpc error: code = NotFound desc = could not find container \"038c8575ff4c08b1387286e23a81f7a935a3d7789ca912b378a0888c7ef5b062\": container with ID starting with 038c8575ff4c08b1387286e23a81f7a935a3d7789ca912b378a0888c7ef5b062 not found: ID does not exist"
Jul 17 19:24:27.000130 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.000111    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"]
Jul 17 19:24:27.001352 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.001333    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/794ccbd1-2557-4fc1-ba9d-a6783232f01a-dshm\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:24:27.001476 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.001356    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/794ccbd1-2557-4fc1-ba9d-a6783232f01a-tls-certs\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:24:27.001476 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.001370    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d4tr\" (UniqueName: \"kubernetes.io/projected/794ccbd1-2557-4fc1-ba9d-a6783232f01a-kube-api-access-9d4tr\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:24:27.004101 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.004077    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4aee7c/test-model-4aee7c-kserve-79b74ff7f9-569km"]
Jul 17 19:24:27.357281 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.357240    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28" podUID="01d0d0c4-bade-428d-a697-a71aa6875663" containerName="main" containerID="cri-o://21061cb38ce7188530fd7f952bbb747bf8551c5ba8b7bbdd5bab6c880662f6ce" gracePeriod=45
Jul 17 19:24:27.615160 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.615090    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:24:27.707347 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.707297    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-tmp-dir\") pod \"01d0d0c4-bade-428d-a697-a71aa6875663\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") "
Jul 17 19:24:27.707347 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.707351    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-model-cache\") pod \"01d0d0c4-bade-428d-a697-a71aa6875663\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") "
Jul 17 19:24:27.707653 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.707386    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-dshm\") pod \"01d0d0c4-bade-428d-a697-a71aa6875663\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") "
Jul 17 19:24:27.707653 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.707530    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5nlb\" (UniqueName: \"kubernetes.io/projected/01d0d0c4-bade-428d-a697-a71aa6875663-kube-api-access-z5nlb\") pod \"01d0d0c4-bade-428d-a697-a71aa6875663\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") "
Jul 17 19:24:27.707653 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.707569    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "01d0d0c4-bade-428d-a697-a71aa6875663" (UID: "01d0d0c4-bade-428d-a697-a71aa6875663"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:24:27.707653 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.707622    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-home\") pod \"01d0d0c4-bade-428d-a697-a71aa6875663\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") "
Jul 17 19:24:27.707885 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.707650    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-model-cache" (OuterVolumeSpecName: "model-cache") pod "01d0d0c4-bade-428d-a697-a71aa6875663" (UID: "01d0d0c4-bade-428d-a697-a71aa6875663"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:24:27.707885 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.707682    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01d0d0c4-bade-428d-a697-a71aa6875663-tls-certs\") pod \"01d0d0c4-bade-428d-a697-a71aa6875663\" (UID: \"01d0d0c4-bade-428d-a697-a71aa6875663\") "
Jul 17 19:24:27.707885 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.707856    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-home" (OuterVolumeSpecName: "home") pod "01d0d0c4-bade-428d-a697-a71aa6875663" (UID: "01d0d0c4-bade-428d-a697-a71aa6875663"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:24:27.708117 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.708100    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-home\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:24:27.708176 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.708123    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-tmp-dir\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:24:27.708176 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.708137    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-model-cache\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:24:27.710036 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.710010    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01d0d0c4-bade-428d-a697-a71aa6875663-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01d0d0c4-bade-428d-a697-a71aa6875663" (UID: "01d0d0c4-bade-428d-a697-a71aa6875663"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:24:27.710219 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.710201    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-dshm" (OuterVolumeSpecName: "dshm") pod "01d0d0c4-bade-428d-a697-a71aa6875663" (UID: "01d0d0c4-bade-428d-a697-a71aa6875663"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:24:27.710284 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.710238    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d0d0c4-bade-428d-a697-a71aa6875663-kube-api-access-z5nlb" (OuterVolumeSpecName: "kube-api-access-z5nlb") pod "01d0d0c4-bade-428d-a697-a71aa6875663" (UID: "01d0d0c4-bade-428d-a697-a71aa6875663"). InnerVolumeSpecName "kube-api-access-z5nlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:24:27.808633 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.808605    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01d0d0c4-bade-428d-a697-a71aa6875663-dshm\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:24:27.808633 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.808633    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5nlb\" (UniqueName: \"kubernetes.io/projected/01d0d0c4-bade-428d-a697-a71aa6875663-kube-api-access-z5nlb\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:24:27.808633 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.808644    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01d0d0c4-bade-428d-a697-a71aa6875663-tls-certs\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:24:27.983681 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.983595    2559 generic.go:356] "Generic (PLEG): container finished" podID="01d0d0c4-bade-428d-a697-a71aa6875663" containerID="21061cb38ce7188530fd7f952bbb747bf8551c5ba8b7bbdd5bab6c880662f6ce" exitCode=0
Jul 17 19:24:27.983681 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.983641    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"
Jul 17 19:24:27.983925 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.983683    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28" event={"ID":"01d0d0c4-bade-428d-a697-a71aa6875663","Type":"ContainerDied","Data":"21061cb38ce7188530fd7f952bbb747bf8551c5ba8b7bbdd5bab6c880662f6ce"}
Jul 17 19:24:27.983925 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.983726    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28" event={"ID":"01d0d0c4-bade-428d-a697-a71aa6875663","Type":"ContainerDied","Data":"9785e75d10d8df7369c6ee796c7f42ca1524415e5ea8d540ea70751710b96e49"}
Jul 17 19:24:27.983925 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.983752    2559 scope.go:117] "RemoveContainer" containerID="21061cb38ce7188530fd7f952bbb747bf8551c5ba8b7bbdd5bab6c880662f6ce"
Jul 17 19:24:27.993619 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.993600    2559 scope.go:117] "RemoveContainer" containerID="21061cb38ce7188530fd7f952bbb747bf8551c5ba8b7bbdd5bab6c880662f6ce"
Jul 17 19:24:27.993861 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:24:27.993842    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21061cb38ce7188530fd7f952bbb747bf8551c5ba8b7bbdd5bab6c880662f6ce\": container with ID starting with 21061cb38ce7188530fd7f952bbb747bf8551c5ba8b7bbdd5bab6c880662f6ce not found: ID does not exist" containerID="21061cb38ce7188530fd7f952bbb747bf8551c5ba8b7bbdd5bab6c880662f6ce"
Jul 17 19:24:27.993900 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:27.993869    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21061cb38ce7188530fd7f952bbb747bf8551c5ba8b7bbdd5bab6c880662f6ce"} err="failed to get container status \"21061cb38ce7188530fd7f952bbb747bf8551c5ba8b7bbdd5bab6c880662f6ce\": rpc error: code = NotFound desc = could not find container \"21061cb38ce7188530fd7f952bbb747bf8551c5ba8b7bbdd5bab6c880662f6ce\": container with ID starting with 21061cb38ce7188530fd7f952bbb747bf8551c5ba8b7bbdd5bab6c880662f6ce not found: ID does not exist"
Jul 17 19:24:28.007238 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:28.007217    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"]
Jul 17 19:24:28.013376 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:28.013357    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-15f29d/test-model-15f29d-kserve-786977d759-r9g28"]
Jul 17 19:24:28.125557 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:28.125523    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01d0d0c4-bade-428d-a697-a71aa6875663" path="/var/lib/kubelet/pods/01d0d0c4-bade-428d-a697-a71aa6875663/volumes"
Jul 17 19:24:28.125888 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:28.125875    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="794ccbd1-2557-4fc1-ba9d-a6783232f01a" path="/var/lib/kubelet/pods/794ccbd1-2557-4fc1-ba9d-a6783232f01a/volumes"
Jul 17 19:24:31.324036 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324005    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq"]
Jul 17 19:24:31.324587 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324570    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="febb45f9-0b54-4579-ac94-759fbd997aa1" containerName="revoke-keys"
Jul 17 19:24:31.324643 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324594    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="febb45f9-0b54-4579-ac94-759fbd997aa1" containerName="revoke-keys"
Jul 17 19:24:31.324643 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324606    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="febb45f9-0b54-4579-ac94-759fbd997aa1" containerName="revoke-keys"
Jul 17 19:24:31.324643 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324616    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="febb45f9-0b54-4579-ac94-759fbd997aa1" containerName="revoke-keys"
Jul 17 19:24:31.324643 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324635    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df312190-1533-4d48-b790-701bba98cae8" containerName="revoke-keys"
Jul 17 19:24:31.324643 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324642    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="df312190-1533-4d48-b790-701bba98cae8" containerName="revoke-keys"
Jul 17 19:24:31.324813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324657    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df312190-1533-4d48-b790-701bba98cae8" containerName="revoke-keys"
Jul 17 19:24:31.324813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324665    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="df312190-1533-4d48-b790-701bba98cae8" containerName="revoke-keys"
Jul 17 19:24:31.324813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324675    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df312190-1533-4d48-b790-701bba98cae8" containerName="revoke-keys"
Jul 17 19:24:31.324813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324680    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="df312190-1533-4d48-b790-701bba98cae8" containerName="revoke-keys"
Jul 17 19:24:31.324813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324689    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="794ccbd1-2557-4fc1-ba9d-a6783232f01a" containerName="main"
Jul 17 19:24:31.324813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324693    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="794ccbd1-2557-4fc1-ba9d-a6783232f01a" containerName="main"
Jul 17 19:24:31.324813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324709    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01d0d0c4-bade-428d-a697-a71aa6875663" containerName="main"
Jul 17 19:24:31.324813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324713    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="01d0d0c4-bade-428d-a697-a71aa6875663" containerName="main"
Jul 17 19:24:31.324813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324779    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="df312190-1533-4d48-b790-701bba98cae8" containerName="revoke-keys"
Jul 17 19:24:31.324813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324786    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="794ccbd1-2557-4fc1-ba9d-a6783232f01a" containerName="main"
Jul 17 19:24:31.324813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324791    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="febb45f9-0b54-4579-ac94-759fbd997aa1" containerName="revoke-keys"
Jul 17 19:24:31.324813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324797    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="febb45f9-0b54-4579-ac94-759fbd997aa1" containerName="revoke-keys"
Jul 17 19:24:31.324813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324802    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="febb45f9-0b54-4579-ac94-759fbd997aa1" containerName="revoke-keys"
Jul 17 19:24:31.324813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324809    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="01d0d0c4-bade-428d-a697-a71aa6875663" containerName="main"
Jul 17 19:24:31.324813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.324814    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="df312190-1533-4d48-b790-701bba98cae8" containerName="revoke-keys"
Jul 17 19:24:31.328850 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.328835    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq"
Jul 17 19:24:31.331322 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.331286    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:24:31.334900 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.334880    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq"]
Jul 17 19:24:31.438429 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.438398    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d0df588-8030-4fdf-897f-a3c0bcc33e51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-x47rq\" (UID: \"9d0df588-8030-4fdf-897f-a3c0bcc33e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq"
Jul 17 19:24:31.539060 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.539029    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d0df588-8030-4fdf-897f-a3c0bcc33e51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-x47rq\" (UID: \"9d0df588-8030-4fdf-897f-a3c0bcc33e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq"
Jul 17 19:24:31.539727 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.539702    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d0df588-8030-4fdf-897f-a3c0bcc33e51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-x47rq\" (UID: \"9d0df588-8030-4fdf-897f-a3c0bcc33e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq"
Jul 17 19:24:31.640945 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.640854    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq"
Jul 17 19:24:31.773513 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:31.773488    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq"]
Jul 17 19:24:31.774700 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:24:31.774674    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0df588_8030_4fdf_897f_a3c0bcc33e51.slice/crio-4b1e9c283fd9a12925644c2649cf00e7e82e5b5f09822a9bee21a1817a910613 WatchSource:0}: Error finding container 4b1e9c283fd9a12925644c2649cf00e7e82e5b5f09822a9bee21a1817a910613: Status 404 returned error can't find the container with id 4b1e9c283fd9a12925644c2649cf00e7e82e5b5f09822a9bee21a1817a910613
Jul 17 19:24:32.009101 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:32.009068    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq" event={"ID":"9d0df588-8030-4fdf-897f-a3c0bcc33e51","Type":"ContainerStarted","Data":"4b1e9c283fd9a12925644c2649cf00e7e82e5b5f09822a9bee21a1817a910613"}
Jul 17 19:24:33.016369 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:33.016327    2559 generic.go:356] "Generic (PLEG): container finished" podID="9d0df588-8030-4fdf-897f-a3c0bcc33e51" containerID="b96fa578b29af3cd4a6683657f8582a869268ac0b57a1b5c58c9c159cd81872b" exitCode=6
Jul 17 19:24:33.016751 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:33.016422    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq" event={"ID":"9d0df588-8030-4fdf-897f-a3c0bcc33e51","Type":"ContainerDied","Data":"b96fa578b29af3cd4a6683657f8582a869268ac0b57a1b5c58c9c159cd81872b"}
Jul 17 19:24:33.016751 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:33.016626    2559 scope.go:117] "RemoveContainer" containerID="b96fa578b29af3cd4a6683657f8582a869268ac0b57a1b5c58c9c159cd81872b"
Jul 17 19:24:34.023971 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:34.023881    2559 generic.go:356] "Generic (PLEG): container finished" podID="9d0df588-8030-4fdf-897f-a3c0bcc33e51" containerID="b3eb72e4d202b6e8c413878ac0f93508936add3a8db4cf97807af097ab7649f8" exitCode=6
Jul 17 19:24:34.024414 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:34.023968    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq" event={"ID":"9d0df588-8030-4fdf-897f-a3c0bcc33e51","Type":"ContainerDied","Data":"b3eb72e4d202b6e8c413878ac0f93508936add3a8db4cf97807af097ab7649f8"}
Jul 17 19:24:34.024414 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:34.024040    2559 scope.go:117] "RemoveContainer" containerID="b96fa578b29af3cd4a6683657f8582a869268ac0b57a1b5c58c9c159cd81872b"
Jul 17 19:24:34.024414 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:34.024204    2559 scope.go:117] "RemoveContainer" containerID="b3eb72e4d202b6e8c413878ac0f93508936add3a8db4cf97807af097ab7649f8"
Jul 17 19:24:34.024527 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:24:34.024499    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-x47rq_odh-ai-gateway-infra(9d0df588-8030-4fdf-897f-a3c0bcc33e51)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq" podUID="9d0df588-8030-4fdf-897f-a3c0bcc33e51"
Jul 17 19:24:35.031539 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:35.031511    2559 scope.go:117] "RemoveContainer" containerID="b3eb72e4d202b6e8c413878ac0f93508936add3a8db4cf97807af097ab7649f8"
Jul 17 19:24:35.031933 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:24:35.031718    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-x47rq_odh-ai-gateway-infra(9d0df588-8030-4fdf-897f-a3c0bcc33e51)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq" podUID="9d0df588-8030-4fdf-897f-a3c0bcc33e51"
Jul 17 19:24:35.995976 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:35.995937    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7"]
Jul 17 19:24:35.996375 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:35.996360    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="febb45f9-0b54-4579-ac94-759fbd997aa1" containerName="revoke-keys"
Jul 17 19:24:35.996452 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:35.996377    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="febb45f9-0b54-4579-ac94-759fbd997aa1" containerName="revoke-keys"
Jul 17 19:24:35.996512 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:35.996484    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="df312190-1533-4d48-b790-701bba98cae8" containerName="revoke-keys"
Jul 17 19:24:36.000828 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:36.000811    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7"
Jul 17 19:24:36.006755 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:36.006725    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7"]
Jul 17 19:24:36.080549 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:36.080519    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef36f08-dab6-4907-9798-1369505f7161-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7\" (UID: \"0ef36f08-dab6-4907-9798-1369505f7161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7"
Jul 17 19:24:36.181465 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:36.181433    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef36f08-dab6-4907-9798-1369505f7161-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7\" (UID: \"0ef36f08-dab6-4907-9798-1369505f7161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7"
Jul 17 19:24:36.181991 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:36.181972    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef36f08-dab6-4907-9798-1369505f7161-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7\" (UID: \"0ef36f08-dab6-4907-9798-1369505f7161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7"
Jul 17 19:24:36.313390 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:36.313362    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7"
Jul 17 19:24:36.447876 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:36.447853    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7"]
Jul 17 19:24:36.449437 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:24:36.449411    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef36f08_dab6_4907_9798_1369505f7161.slice/crio-a5482984734250f74efd995d0e51e8bcd24f53804495deb4869caf9b089f8729 WatchSource:0}: Error finding container a5482984734250f74efd995d0e51e8bcd24f53804495deb4869caf9b089f8729: Status 404 returned error can't find the container with id a5482984734250f74efd995d0e51e8bcd24f53804495deb4869caf9b089f8729
Jul 17 19:24:37.044200 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:37.044112    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7" event={"ID":"0ef36f08-dab6-4907-9798-1369505f7161","Type":"ContainerStarted","Data":"cf12180d24af3036811e03bfd1a8f195f380b245f02c955622efb4d045dd365e"}
Jul 17 19:24:37.044200 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:37.044163    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7" event={"ID":"0ef36f08-dab6-4907-9798-1369505f7161","Type":"ContainerStarted","Data":"a5482984734250f74efd995d0e51e8bcd24f53804495deb4869caf9b089f8729"}
Jul 17 19:24:37.060373 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:37.060324    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7" podStartSLOduration=2.060294452 podStartE2EDuration="2.060294452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:24:37.058510205 +0000 UTC m=+3087.500018895" watchObservedRunningTime="2026-07-17 19:24:37.060294452 +0000 UTC m=+3087.501803141"
Jul 17 19:24:38.051100 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:38.051063    2559 generic.go:356] "Generic (PLEG): container finished" podID="0ef36f08-dab6-4907-9798-1369505f7161" containerID="cf12180d24af3036811e03bfd1a8f195f380b245f02c955622efb4d045dd365e" exitCode=6
Jul 17 19:24:38.051553 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:38.051151    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7" event={"ID":"0ef36f08-dab6-4907-9798-1369505f7161","Type":"ContainerDied","Data":"cf12180d24af3036811e03bfd1a8f195f380b245f02c955622efb4d045dd365e"}
Jul 17 19:24:38.051553 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:38.051334    2559 scope.go:117] "RemoveContainer" containerID="cf12180d24af3036811e03bfd1a8f195f380b245f02c955622efb4d045dd365e"
Jul 17 19:24:38.610320 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:38.610224    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15f29d-7m5wl"]
Jul 17 19:24:38.614073 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:38.614056    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15f29d-7m5wl"
Jul 17 19:24:38.620809 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:38.620729    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15f29d-7m5wl"]
Jul 17 19:24:38.699527 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:38.699494    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe5ba99a-ce82-48e2-9f58-ce45b0e2315a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-15f29d-7m5wl\" (UID: \"fe5ba99a-ce82-48e2-9f58-ce45b0e2315a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15f29d-7m5wl"
Jul 17 19:24:38.800519 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:38.800480    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe5ba99a-ce82-48e2-9f58-ce45b0e2315a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-15f29d-7m5wl\" (UID: \"fe5ba99a-ce82-48e2-9f58-ce45b0e2315a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15f29d-7m5wl"
Jul 17 19:24:38.801078 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:38.801057    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe5ba99a-ce82-48e2-9f58-ce45b0e2315a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-15f29d-7m5wl\" (UID: \"fe5ba99a-ce82-48e2-9f58-ce45b0e2315a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15f29d-7m5wl"
Jul 17 19:24:38.926629 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:38.926547    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15f29d-7m5wl"
Jul 17 19:24:39.058806 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:39.058777    2559 generic.go:356] "Generic (PLEG): container finished" podID="0ef36f08-dab6-4907-9798-1369505f7161" containerID="a79a561bdf97a25394ce55ce2777e74598c8ce1f0b2d6c61903c5c0c8e020b46" exitCode=6
Jul 17 19:24:39.059179 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:39.058859    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7" event={"ID":"0ef36f08-dab6-4907-9798-1369505f7161","Type":"ContainerDied","Data":"a79a561bdf97a25394ce55ce2777e74598c8ce1f0b2d6c61903c5c0c8e020b46"}
Jul 17 19:24:39.059179 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:39.058920    2559 scope.go:117] "RemoveContainer" containerID="cf12180d24af3036811e03bfd1a8f195f380b245f02c955622efb4d045dd365e"
Jul 17 19:24:39.059179 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:39.059124    2559 scope.go:117] "RemoveContainer" containerID="a79a561bdf97a25394ce55ce2777e74598c8ce1f0b2d6c61903c5c0c8e020b46"
Jul 17 19:24:39.059380 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:24:39.059362    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7_odh-ai-gateway-infra(0ef36f08-dab6-4907-9798-1369505f7161)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7" podUID="0ef36f08-dab6-4907-9798-1369505f7161"
Jul 17 19:24:39.064552 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:39.064531    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15f29d-7m5wl"]
Jul 17 19:24:39.066023 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:24:39.065995    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5ba99a_ce82_48e2_9f58_ce45b0e2315a.slice/crio-1fce2f9628c29da44534c5ab5000c0308fc47ab163c55f090859a9766a7a023c WatchSource:0}: Error finding container 1fce2f9628c29da44534c5ab5000c0308fc47ab163c55f090859a9766a7a023c: Status 404 returned error can't find the container with id 1fce2f9628c29da44534c5ab5000c0308fc47ab163c55f090859a9766a7a023c
Jul 17 19:24:40.066885 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:40.066846    2559 scope.go:117] "RemoveContainer" containerID="a79a561bdf97a25394ce55ce2777e74598c8ce1f0b2d6c61903c5c0c8e020b46"
Jul 17 19:24:40.067280 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:24:40.067073    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7_odh-ai-gateway-infra(0ef36f08-dab6-4907-9798-1369505f7161)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7" podUID="0ef36f08-dab6-4907-9798-1369505f7161"
Jul 17 19:24:40.068114 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:40.068093    2559 generic.go:356] "Generic (PLEG): container finished" podID="fe5ba99a-ce82-48e2-9f58-ce45b0e2315a" containerID="032bf248cd073f68e916055cfc650173c8e0c077f5cadd43559519643d2da2e6" exitCode=0
Jul 17 19:24:40.068186 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:40.068148    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15f29d-7m5wl" event={"ID":"fe5ba99a-ce82-48e2-9f58-ce45b0e2315a","Type":"ContainerDied","Data":"032bf248cd073f68e916055cfc650173c8e0c077f5cadd43559519643d2da2e6"}
Jul 17 19:24:40.068220 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:40.068190    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15f29d-7m5wl" event={"ID":"fe5ba99a-ce82-48e2-9f58-ce45b0e2315a","Type":"ContainerStarted","Data":"1fce2f9628c29da44534c5ab5000c0308fc47ab163c55f090859a9766a7a023c"}
Jul 17 19:24:41.206808 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:41.206781    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15f29d-7m5wl"
Jul 17 19:24:41.226732 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:41.226690    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe5ba99a-ce82-48e2-9f58-ce45b0e2315a-maas-api-service-ca\") pod \"fe5ba99a-ce82-48e2-9f58-ce45b0e2315a\" (UID: \"fe5ba99a-ce82-48e2-9f58-ce45b0e2315a\") "
Jul 17 19:24:41.227106 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:41.227010    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5ba99a-ce82-48e2-9f58-ce45b0e2315a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe5ba99a-ce82-48e2-9f58-ce45b0e2315a" (UID: "fe5ba99a-ce82-48e2-9f58-ce45b0e2315a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:24:41.328490 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:41.328419    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe5ba99a-ce82-48e2-9f58-ce45b0e2315a-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:24:42.081148 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:42.081122    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15f29d-7m5wl"
Jul 17 19:24:42.081376 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:42.081121    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15f29d-7m5wl" event={"ID":"fe5ba99a-ce82-48e2-9f58-ce45b0e2315a","Type":"ContainerDied","Data":"1fce2f9628c29da44534c5ab5000c0308fc47ab163c55f090859a9766a7a023c"}
Jul 17 19:24:42.081376 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:42.081249    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fce2f9628c29da44534c5ab5000c0308fc47ab163c55f090859a9766a7a023c"
Jul 17 19:24:47.119686 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:47.119656    2559 scope.go:117] "RemoveContainer" containerID="b3eb72e4d202b6e8c413878ac0f93508936add3a8db4cf97807af097ab7649f8"
Jul 17 19:24:48.117460 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:48.117429    2559 generic.go:356] "Generic (PLEG): container finished" podID="9d0df588-8030-4fdf-897f-a3c0bcc33e51" containerID="c8ac3eab9b829abd51d4f152bf750a94a208006813d4dca3534b20c3be746a5b" exitCode=6
Jul 17 19:24:48.117611 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:48.117481    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq" event={"ID":"9d0df588-8030-4fdf-897f-a3c0bcc33e51","Type":"ContainerDied","Data":"c8ac3eab9b829abd51d4f152bf750a94a208006813d4dca3534b20c3be746a5b"}
Jul 17 19:24:48.117611 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:48.117529    2559 scope.go:117] "RemoveContainer" containerID="b3eb72e4d202b6e8c413878ac0f93508936add3a8db4cf97807af097ab7649f8"
Jul 17 19:24:48.117850 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:48.117830    2559 scope.go:117] "RemoveContainer" containerID="c8ac3eab9b829abd51d4f152bf750a94a208006813d4dca3534b20c3be746a5b"
Jul 17 19:24:48.118077 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:24:48.118053    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-x47rq_odh-ai-gateway-infra(9d0df588-8030-4fdf-897f-a3c0bcc33e51)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq" podUID="9d0df588-8030-4fdf-897f-a3c0bcc33e51"
Jul 17 19:24:49.147302 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:49.147268    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq"]
Jul 17 19:24:49.280356 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:49.280332    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq"
Jul 17 19:24:49.402201 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:49.402123    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d0df588-8030-4fdf-897f-a3c0bcc33e51-maas-api-service-ca\") pod \"9d0df588-8030-4fdf-897f-a3c0bcc33e51\" (UID: \"9d0df588-8030-4fdf-897f-a3c0bcc33e51\") "
Jul 17 19:24:49.402606 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:49.402579    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0df588-8030-4fdf-897f-a3c0bcc33e51-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9d0df588-8030-4fdf-897f-a3c0bcc33e51" (UID: "9d0df588-8030-4fdf-897f-a3c0bcc33e51"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:24:49.503491 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:49.503457    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d0df588-8030-4fdf-897f-a3c0bcc33e51-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:24:50.131845 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:50.131814    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq"
Jul 17 19:24:50.132025 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:50.131881    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq" event={"ID":"9d0df588-8030-4fdf-897f-a3c0bcc33e51","Type":"ContainerDied","Data":"4b1e9c283fd9a12925644c2649cf00e7e82e5b5f09822a9bee21a1817a910613"}
Jul 17 19:24:50.132025 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:50.131932    2559 scope.go:117] "RemoveContainer" containerID="c8ac3eab9b829abd51d4f152bf750a94a208006813d4dca3534b20c3be746a5b"
Jul 17 19:24:50.149210 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:50.149187    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq"]
Jul 17 19:24:50.153006 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:50.152985    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-x47rq"]
Jul 17 19:24:52.125510 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:52.125475    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d0df588-8030-4fdf-897f-a3c0bcc33e51" path="/var/lib/kubelet/pods/9d0df588-8030-4fdf-897f-a3c0bcc33e51/volumes"
Jul 17 19:24:53.119463 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:53.119434    2559 scope.go:117] "RemoveContainer" containerID="a79a561bdf97a25394ce55ce2777e74598c8ce1f0b2d6c61903c5c0c8e020b46"
Jul 17 19:24:54.156029 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:54.155993    2559 generic.go:356] "Generic (PLEG): container finished" podID="0ef36f08-dab6-4907-9798-1369505f7161" containerID="601f5475d488009b40f222f4ab8ad8e8c9848f4d838108e2e9622481deaabb98" exitCode=6
Jul 17 19:24:54.156477 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:54.156062    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7" event={"ID":"0ef36f08-dab6-4907-9798-1369505f7161","Type":"ContainerDied","Data":"601f5475d488009b40f222f4ab8ad8e8c9848f4d838108e2e9622481deaabb98"}
Jul 17 19:24:54.156477 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:54.156123    2559 scope.go:117] "RemoveContainer" containerID="a79a561bdf97a25394ce55ce2777e74598c8ce1f0b2d6c61903c5c0c8e020b46"
Jul 17 19:24:54.156477 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:54.156465    2559 scope.go:117] "RemoveContainer" containerID="601f5475d488009b40f222f4ab8ad8e8c9848f4d838108e2e9622481deaabb98"
Jul 17 19:24:54.156678 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:24:54.156659    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7_odh-ai-gateway-infra(0ef36f08-dab6-4907-9798-1369505f7161)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7" podUID="0ef36f08-dab6-4907-9798-1369505f7161"
Jul 17 19:24:55.186890 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:55.186857    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7"]
Jul 17 19:24:55.328190 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:55.328164    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7"
Jul 17 19:24:55.462535 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:55.462455    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef36f08-dab6-4907-9798-1369505f7161-maas-api-service-ca\") pod \"0ef36f08-dab6-4907-9798-1369505f7161\" (UID: \"0ef36f08-dab6-4907-9798-1369505f7161\") "
Jul 17 19:24:55.462833 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:55.462809    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ef36f08-dab6-4907-9798-1369505f7161-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ef36f08-dab6-4907-9798-1369505f7161" (UID: "0ef36f08-dab6-4907-9798-1369505f7161"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:24:55.564109 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:55.564078    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef36f08-dab6-4907-9798-1369505f7161-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:24:56.170946 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:56.170921    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7"
Jul 17 19:24:56.171121 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:56.170961    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7" event={"ID":"0ef36f08-dab6-4907-9798-1369505f7161","Type":"ContainerDied","Data":"a5482984734250f74efd995d0e51e8bcd24f53804495deb4869caf9b089f8729"}
Jul 17 19:24:56.171121 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:56.171029    2559 scope.go:117] "RemoveContainer" containerID="601f5475d488009b40f222f4ab8ad8e8c9848f4d838108e2e9622481deaabb98"
Jul 17 19:24:56.187590 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:56.187563    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7"]
Jul 17 19:24:56.193510 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:56.193486    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-n5qf7"]
Jul 17 19:24:58.126472 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:24:58.126439    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ef36f08-dab6-4907-9798-1369505f7161" path="/var/lib/kubelet/pods/0ef36f08-dab6-4907-9798-1369505f7161/volumes"
Jul 17 19:25:21.363551 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.363509    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw"]
Jul 17 19:25:21.364100 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.364078    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe5ba99a-ce82-48e2-9f58-ce45b0e2315a" containerName="revoke-keys"
Jul 17 19:25:21.364187 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.364103    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe5ba99a-ce82-48e2-9f58-ce45b0e2315a" containerName="revoke-keys"
Jul 17 19:25:21.364187 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.364117    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ef36f08-dab6-4907-9798-1369505f7161" containerName="revoke-keys"
Jul 17 19:25:21.364187 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.364124    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ef36f08-dab6-4907-9798-1369505f7161" containerName="revoke-keys"
Jul 17 19:25:21.364187 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.364142    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ef36f08-dab6-4907-9798-1369505f7161" containerName="revoke-keys"
Jul 17 19:25:21.364187 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.364151    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ef36f08-dab6-4907-9798-1369505f7161" containerName="revoke-keys"
Jul 17 19:25:21.364187 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.364164    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0df588-8030-4fdf-897f-a3c0bcc33e51" containerName="revoke-keys"
Jul 17 19:25:21.364187 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.364172    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0df588-8030-4fdf-897f-a3c0bcc33e51" containerName="revoke-keys"
Jul 17 19:25:21.364187 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.364187    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0df588-8030-4fdf-897f-a3c0bcc33e51" containerName="revoke-keys"
Jul 17 19:25:21.364608 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.364196    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0df588-8030-4fdf-897f-a3c0bcc33e51" containerName="revoke-keys"
Jul 17 19:25:21.364608 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.364206    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ef36f08-dab6-4907-9798-1369505f7161" containerName="revoke-keys"
Jul 17 19:25:21.364608 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.364215    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ef36f08-dab6-4907-9798-1369505f7161" containerName="revoke-keys"
Jul 17 19:25:21.364608 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.364343    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d0df588-8030-4fdf-897f-a3c0bcc33e51" containerName="revoke-keys"
Jul 17 19:25:21.364608 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.364356    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d0df588-8030-4fdf-897f-a3c0bcc33e51" containerName="revoke-keys"
Jul 17 19:25:21.364608 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.364369    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ef36f08-dab6-4907-9798-1369505f7161" containerName="revoke-keys"
Jul 17 19:25:21.364608 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.364378    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ef36f08-dab6-4907-9798-1369505f7161" containerName="revoke-keys"
Jul 17 19:25:21.364608 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.364385    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe5ba99a-ce82-48e2-9f58-ce45b0e2315a" containerName="revoke-keys"
Jul 17 19:25:21.364608 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.364397    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ef36f08-dab6-4907-9798-1369505f7161" containerName="revoke-keys"
Jul 17 19:25:21.367664 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.367643    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw"
Jul 17 19:25:21.370086 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.370068    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:25:21.375638 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.375241    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw"]
Jul 17 19:25:21.510896 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.510859    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1535dc-ec0a-4743-a2fc-f8848558395c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw\" (UID: \"2f1535dc-ec0a-4743-a2fc-f8848558395c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw"
Jul 17 19:25:21.611770 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.611745    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1535dc-ec0a-4743-a2fc-f8848558395c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw\" (UID: \"2f1535dc-ec0a-4743-a2fc-f8848558395c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw"
Jul 17 19:25:21.612341 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.612301    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1535dc-ec0a-4743-a2fc-f8848558395c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw\" (UID: \"2f1535dc-ec0a-4743-a2fc-f8848558395c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw"
Jul 17 19:25:21.678838 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.678785    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw"
Jul 17 19:25:21.811247 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:21.811224    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw"]
Jul 17 19:25:21.812878 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:25:21.812852    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1535dc_ec0a_4743_a2fc_f8848558395c.slice/crio-abe64893f1a76964459d33bcff3dfd6b0e7b8e7f2699345768f25d0232de6870 WatchSource:0}: Error finding container abe64893f1a76964459d33bcff3dfd6b0e7b8e7f2699345768f25d0232de6870: Status 404 returned error can't find the container with id abe64893f1a76964459d33bcff3dfd6b0e7b8e7f2699345768f25d0232de6870
Jul 17 19:25:22.324991 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:22.324956    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw" event={"ID":"2f1535dc-ec0a-4743-a2fc-f8848558395c","Type":"ContainerStarted","Data":"d3f8de70d09b90002ee31ef9e215a1b7e56fb0892c9ddcc3704dcc201d24cb07"}
Jul 17 19:25:22.325203 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:22.325010    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw" event={"ID":"2f1535dc-ec0a-4743-a2fc-f8848558395c","Type":"ContainerStarted","Data":"abe64893f1a76964459d33bcff3dfd6b0e7b8e7f2699345768f25d0232de6870"}
Jul 17 19:25:22.340364 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:22.340284    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw" podStartSLOduration=1.3402649979999999 podStartE2EDuration="1.340264998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:25:22.339812267 +0000 UTC m=+3132.781320957" watchObservedRunningTime="2026-07-17 19:25:22.340264998 +0000 UTC m=+3132.781773703"
Jul 17 19:25:23.331299 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:23.331261    2559 generic.go:356] "Generic (PLEG): container finished" podID="2f1535dc-ec0a-4743-a2fc-f8848558395c" containerID="d3f8de70d09b90002ee31ef9e215a1b7e56fb0892c9ddcc3704dcc201d24cb07" exitCode=6
Jul 17 19:25:23.331760 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:23.331343    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw" event={"ID":"2f1535dc-ec0a-4743-a2fc-f8848558395c","Type":"ContainerDied","Data":"d3f8de70d09b90002ee31ef9e215a1b7e56fb0892c9ddcc3704dcc201d24cb07"}
Jul 17 19:25:23.331760 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:23.331534    2559 scope.go:117] "RemoveContainer" containerID="d3f8de70d09b90002ee31ef9e215a1b7e56fb0892c9ddcc3704dcc201d24cb07"
Jul 17 19:25:24.339430 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:24.339402    2559 generic.go:356] "Generic (PLEG): container finished" podID="2f1535dc-ec0a-4743-a2fc-f8848558395c" containerID="ddd2ec60ada23d2374f92c7b550873f56831a1ed818b7f0cc1b6f4624b051d9f" exitCode=6
Jul 17 19:25:24.339847 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:24.339472    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw" event={"ID":"2f1535dc-ec0a-4743-a2fc-f8848558395c","Type":"ContainerDied","Data":"ddd2ec60ada23d2374f92c7b550873f56831a1ed818b7f0cc1b6f4624b051d9f"}
Jul 17 19:25:24.339847 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:24.339512    2559 scope.go:117] "RemoveContainer" containerID="d3f8de70d09b90002ee31ef9e215a1b7e56fb0892c9ddcc3704dcc201d24cb07"
Jul 17 19:25:24.339847 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:24.339707    2559 scope.go:117] "RemoveContainer" containerID="ddd2ec60ada23d2374f92c7b550873f56831a1ed818b7f0cc1b6f4624b051d9f"
Jul 17 19:25:24.340006 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:25:24.339940    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw_odh-ai-gateway-infra(2f1535dc-ec0a-4743-a2fc-f8848558395c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw" podUID="2f1535dc-ec0a-4743-a2fc-f8848558395c"
Jul 17 19:25:25.346921 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:25.346896    2559 scope.go:117] "RemoveContainer" containerID="ddd2ec60ada23d2374f92c7b550873f56831a1ed818b7f0cc1b6f4624b051d9f"
Jul 17 19:25:25.347326 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:25:25.347065    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw_odh-ai-gateway-infra(2f1535dc-ec0a-4743-a2fc-f8848558395c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw" podUID="2f1535dc-ec0a-4743-a2fc-f8848558395c"
Jul 17 19:25:28.362756 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:28.362727    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40565f-wthwq"]
Jul 17 19:25:28.363301 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:28.363281    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0df588-8030-4fdf-897f-a3c0bcc33e51" containerName="revoke-keys"
Jul 17 19:25:28.363301 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:28.363322    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0df588-8030-4fdf-897f-a3c0bcc33e51" containerName="revoke-keys"
Jul 17 19:25:28.363473 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:28.363464    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d0df588-8030-4fdf-897f-a3c0bcc33e51" containerName="revoke-keys"
Jul 17 19:25:28.367820 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:28.367799    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40565f-wthwq"
Jul 17 19:25:28.374500 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:28.374460    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40565f-wthwq"]
Jul 17 19:25:28.462900 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:28.462875    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f0d0c4a-928a-488b-be6d-0c274a3c604a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-40565f-wthwq\" (UID: \"2f0d0c4a-928a-488b-be6d-0c274a3c604a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40565f-wthwq"
Jul 17 19:25:28.563459 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:28.563429    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f0d0c4a-928a-488b-be6d-0c274a3c604a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-40565f-wthwq\" (UID: \"2f0d0c4a-928a-488b-be6d-0c274a3c604a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40565f-wthwq"
Jul 17 19:25:28.563981 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:28.563960    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f0d0c4a-928a-488b-be6d-0c274a3c604a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-40565f-wthwq\" (UID: \"2f0d0c4a-928a-488b-be6d-0c274a3c604a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40565f-wthwq"
Jul 17 19:25:28.680289 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:28.680219    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40565f-wthwq"
Jul 17 19:25:29.027337 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:29.027277    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40565f-wthwq"]
Jul 17 19:25:29.029638 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:25:29.029612    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0d0c4a_928a_488b_be6d_0c274a3c604a.slice/crio-78bd2e6215507015e820090cf7f62f5a6a769edfac0670a84f964792c41484f5 WatchSource:0}: Error finding container 78bd2e6215507015e820090cf7f62f5a6a769edfac0670a84f964792c41484f5: Status 404 returned error can't find the container with id 78bd2e6215507015e820090cf7f62f5a6a769edfac0670a84f964792c41484f5
Jul 17 19:25:29.371757 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:29.371718    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40565f-wthwq" event={"ID":"2f0d0c4a-928a-488b-be6d-0c274a3c604a","Type":"ContainerStarted","Data":"6f97dc667003dd093a7d025ba8747101267c43499a740ff786ff9cf7c8d84347"}
Jul 17 19:25:29.372215 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:29.371777    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40565f-wthwq" event={"ID":"2f0d0c4a-928a-488b-be6d-0c274a3c604a","Type":"ContainerStarted","Data":"78bd2e6215507015e820090cf7f62f5a6a769edfac0670a84f964792c41484f5"}
Jul 17 19:25:29.388420 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:29.388375    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40565f-wthwq" podStartSLOduration=1.388360039 podStartE2EDuration="1.388360039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:25:29.386111894 +0000 UTC m=+3139.827620600" watchObservedRunningTime="2026-07-17 19:25:29.388360039 +0000 UTC m=+3139.829868738"
Jul 17 19:25:30.379020 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:30.378987    2559 generic.go:356] "Generic (PLEG): container finished" podID="2f0d0c4a-928a-488b-be6d-0c274a3c604a" containerID="6f97dc667003dd093a7d025ba8747101267c43499a740ff786ff9cf7c8d84347" exitCode=0
Jul 17 19:25:30.379432 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:30.379074    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40565f-wthwq" event={"ID":"2f0d0c4a-928a-488b-be6d-0c274a3c604a","Type":"ContainerDied","Data":"6f97dc667003dd093a7d025ba8747101267c43499a740ff786ff9cf7c8d84347"}
Jul 17 19:25:31.512639 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:31.512618    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40565f-wthwq"
Jul 17 19:25:31.586659 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:31.586627    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f0d0c4a-928a-488b-be6d-0c274a3c604a-maas-api-service-ca\") pod \"2f0d0c4a-928a-488b-be6d-0c274a3c604a\" (UID: \"2f0d0c4a-928a-488b-be6d-0c274a3c604a\") "
Jul 17 19:25:31.586975 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:31.586954    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0d0c4a-928a-488b-be6d-0c274a3c604a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2f0d0c4a-928a-488b-be6d-0c274a3c604a" (UID: "2f0d0c4a-928a-488b-be6d-0c274a3c604a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:25:31.687684 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:31.687604    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f0d0c4a-928a-488b-be6d-0c274a3c604a-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:25:32.392189 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:32.392098    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40565f-wthwq"
Jul 17 19:25:32.392189 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:32.392128    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40565f-wthwq" event={"ID":"2f0d0c4a-928a-488b-be6d-0c274a3c604a","Type":"ContainerDied","Data":"78bd2e6215507015e820090cf7f62f5a6a769edfac0670a84f964792c41484f5"}
Jul 17 19:25:32.392189 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:32.392174    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78bd2e6215507015e820090cf7f62f5a6a769edfac0670a84f964792c41484f5"
Jul 17 19:25:34.249750 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:34.249720    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d"]
Jul 17 19:25:34.250200 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:34.250142    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f0d0c4a-928a-488b-be6d-0c274a3c604a" containerName="revoke-keys"
Jul 17 19:25:34.250200 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:34.250156    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f0d0c4a-928a-488b-be6d-0c274a3c604a" containerName="revoke-keys"
Jul 17 19:25:34.250334 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:34.250227    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f0d0c4a-928a-488b-be6d-0c274a3c604a" containerName="revoke-keys"
Jul 17 19:25:34.253261 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:34.253244    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d"
Jul 17 19:25:34.261098 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:34.260680    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d"]
Jul 17 19:25:34.305903 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:34.305876    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de6156c-8aca-46a5-b20a-ab378dcc3ecf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d\" (UID: \"1de6156c-8aca-46a5-b20a-ab378dcc3ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d"
Jul 17 19:25:34.407006 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:34.406970    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de6156c-8aca-46a5-b20a-ab378dcc3ecf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d\" (UID: \"1de6156c-8aca-46a5-b20a-ab378dcc3ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d"
Jul 17 19:25:34.407593 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:34.407573    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de6156c-8aca-46a5-b20a-ab378dcc3ecf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d\" (UID: \"1de6156c-8aca-46a5-b20a-ab378dcc3ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d"
Jul 17 19:25:34.566422 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:34.566397    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d"
Jul 17 19:25:34.712500 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:34.712479    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d"]
Jul 17 19:25:34.714914 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:25:34.714887    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de6156c_8aca_46a5_b20a_ab378dcc3ecf.slice/crio-cdb2ddb96ad026cef076ca4cc43ac74078f7524a95a5ae6f20451e5ba7c01646 WatchSource:0}: Error finding container cdb2ddb96ad026cef076ca4cc43ac74078f7524a95a5ae6f20451e5ba7c01646: Status 404 returned error can't find the container with id cdb2ddb96ad026cef076ca4cc43ac74078f7524a95a5ae6f20451e5ba7c01646
Jul 17 19:25:35.410879 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:35.410853    2559 generic.go:356] "Generic (PLEG): container finished" podID="1de6156c-8aca-46a5-b20a-ab378dcc3ecf" containerID="c9ad365533b4e728b0eecf39d2fe152099638f008bfc8da328666bdfc5098734" exitCode=6
Jul 17 19:25:35.411173 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:35.410898    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d" event={"ID":"1de6156c-8aca-46a5-b20a-ab378dcc3ecf","Type":"ContainerDied","Data":"c9ad365533b4e728b0eecf39d2fe152099638f008bfc8da328666bdfc5098734"}
Jul 17 19:25:35.411173 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:35.410942    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d" event={"ID":"1de6156c-8aca-46a5-b20a-ab378dcc3ecf","Type":"ContainerStarted","Data":"cdb2ddb96ad026cef076ca4cc43ac74078f7524a95a5ae6f20451e5ba7c01646"}
Jul 17 19:25:35.411173 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:35.411119    2559 scope.go:117] "RemoveContainer" containerID="c9ad365533b4e728b0eecf39d2fe152099638f008bfc8da328666bdfc5098734"
Jul 17 19:25:36.039570 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:36.039542    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr"]
Jul 17 19:25:36.048671 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:36.045941    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr"
Jul 17 19:25:36.052926 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:36.052901    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr"]
Jul 17 19:25:36.119192 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:36.119164    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fbe0be1-629c-4881-a8ff-3598b15b7def-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr\" (UID: \"0fbe0be1-629c-4881-a8ff-3598b15b7def\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr"
Jul 17 19:25:36.220286 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:36.220247    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fbe0be1-629c-4881-a8ff-3598b15b7def-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr\" (UID: \"0fbe0be1-629c-4881-a8ff-3598b15b7def\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr"
Jul 17 19:25:36.220892 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:36.220873    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fbe0be1-629c-4881-a8ff-3598b15b7def-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr\" (UID: \"0fbe0be1-629c-4881-a8ff-3598b15b7def\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr"
Jul 17 19:25:36.407920 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:36.407898    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr"
Jul 17 19:25:36.419630 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:36.419599    2559 generic.go:356] "Generic (PLEG): container finished" podID="1de6156c-8aca-46a5-b20a-ab378dcc3ecf" containerID="f41308e3e5641a9084055a3b8d6c0666a7e5bb8dc9f24fdf54374e280a8c1210" exitCode=6
Jul 17 19:25:36.419993 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:36.419669    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d" event={"ID":"1de6156c-8aca-46a5-b20a-ab378dcc3ecf","Type":"ContainerDied","Data":"f41308e3e5641a9084055a3b8d6c0666a7e5bb8dc9f24fdf54374e280a8c1210"}
Jul 17 19:25:36.419993 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:36.419725    2559 scope.go:117] "RemoveContainer" containerID="c9ad365533b4e728b0eecf39d2fe152099638f008bfc8da328666bdfc5098734"
Jul 17 19:25:36.420107 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:36.420054    2559 scope.go:117] "RemoveContainer" containerID="f41308e3e5641a9084055a3b8d6c0666a7e5bb8dc9f24fdf54374e280a8c1210"
Jul 17 19:25:36.420370 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:25:36.420342    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d_odh-ai-gateway-infra(1de6156c-8aca-46a5-b20a-ab378dcc3ecf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d" podUID="1de6156c-8aca-46a5-b20a-ab378dcc3ecf"
Jul 17 19:25:36.540259 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:36.540226    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr"]
Jul 17 19:25:36.541765 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:25:36.541738    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbe0be1_629c_4881_a8ff_3598b15b7def.slice/crio-d49ae14fd71343b7add1fd70413aa6009b5b29d89de35b2d9a4a28c178fe5ddc WatchSource:0}: Error finding container d49ae14fd71343b7add1fd70413aa6009b5b29d89de35b2d9a4a28c178fe5ddc: Status 404 returned error can't find the container with id d49ae14fd71343b7add1fd70413aa6009b5b29d89de35b2d9a4a28c178fe5ddc
Jul 17 19:25:37.119893 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:37.119862    2559 scope.go:117] "RemoveContainer" containerID="ddd2ec60ada23d2374f92c7b550873f56831a1ed818b7f0cc1b6f4624b051d9f"
Jul 17 19:25:37.426988 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:37.426908    2559 generic.go:356] "Generic (PLEG): container finished" podID="0fbe0be1-629c-4881-a8ff-3598b15b7def" containerID="e9c3ee4208409f9fbac982a2e8e22449a030c0635393496c024dcece4362a314" exitCode=6
Jul 17 19:25:37.427404 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:37.426980    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr" event={"ID":"0fbe0be1-629c-4881-a8ff-3598b15b7def","Type":"ContainerDied","Data":"e9c3ee4208409f9fbac982a2e8e22449a030c0635393496c024dcece4362a314"}
Jul 17 19:25:37.427404 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:37.427039    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr" event={"ID":"0fbe0be1-629c-4881-a8ff-3598b15b7def","Type":"ContainerStarted","Data":"d49ae14fd71343b7add1fd70413aa6009b5b29d89de35b2d9a4a28c178fe5ddc"}
Jul 17 19:25:37.427404 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:37.427252    2559 scope.go:117] "RemoveContainer" containerID="e9c3ee4208409f9fbac982a2e8e22449a030c0635393496c024dcece4362a314"
Jul 17 19:25:37.428974 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:37.428954    2559 scope.go:117] "RemoveContainer" containerID="f41308e3e5641a9084055a3b8d6c0666a7e5bb8dc9f24fdf54374e280a8c1210"
Jul 17 19:25:37.429172 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:25:37.429142    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d_odh-ai-gateway-infra(1de6156c-8aca-46a5-b20a-ab378dcc3ecf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d" podUID="1de6156c-8aca-46a5-b20a-ab378dcc3ecf"
Jul 17 19:25:38.436909 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:38.436830    2559 generic.go:356] "Generic (PLEG): container finished" podID="2f1535dc-ec0a-4743-a2fc-f8848558395c" containerID="40e7aa9ee06a0d0ae602df48133626ac38facbdf4f507a82452c14bcb2e8f84a" exitCode=6
Jul 17 19:25:38.439225 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:38.436883    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw" event={"ID":"2f1535dc-ec0a-4743-a2fc-f8848558395c","Type":"ContainerDied","Data":"40e7aa9ee06a0d0ae602df48133626ac38facbdf4f507a82452c14bcb2e8f84a"}
Jul 17 19:25:38.439225 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:38.437518    2559 scope.go:117] "RemoveContainer" containerID="ddd2ec60ada23d2374f92c7b550873f56831a1ed818b7f0cc1b6f4624b051d9f"
Jul 17 19:25:38.439225 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:38.437875    2559 scope.go:117] "RemoveContainer" containerID="40e7aa9ee06a0d0ae602df48133626ac38facbdf4f507a82452c14bcb2e8f84a"
Jul 17 19:25:38.439225 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:25:38.438151    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw_odh-ai-gateway-infra(2f1535dc-ec0a-4743-a2fc-f8848558395c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw" podUID="2f1535dc-ec0a-4743-a2fc-f8848558395c"
Jul 17 19:25:38.440534 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:38.440514    2559 generic.go:356] "Generic (PLEG): container finished" podID="0fbe0be1-629c-4881-a8ff-3598b15b7def" containerID="8a6eec432eafc40b22d7746f7a10c17924fa2c900f84c30538e097da3147ad67" exitCode=6
Jul 17 19:25:38.440616 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:38.440541    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr" event={"ID":"0fbe0be1-629c-4881-a8ff-3598b15b7def","Type":"ContainerDied","Data":"8a6eec432eafc40b22d7746f7a10c17924fa2c900f84c30538e097da3147ad67"}
Jul 17 19:25:38.440763 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:38.440750    2559 scope.go:117] "RemoveContainer" containerID="8a6eec432eafc40b22d7746f7a10c17924fa2c900f84c30538e097da3147ad67"
Jul 17 19:25:38.440929 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:25:38.440911    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr_odh-ai-gateway-infra(0fbe0be1-629c-4881-a8ff-3598b15b7def)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr" podUID="0fbe0be1-629c-4881-a8ff-3598b15b7def"
Jul 17 19:25:38.450265 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:38.450250    2559 scope.go:117] "RemoveContainer" containerID="e9c3ee4208409f9fbac982a2e8e22449a030c0635393496c024dcece4362a314"
Jul 17 19:25:39.447965 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:39.447938    2559 scope.go:117] "RemoveContainer" containerID="8a6eec432eafc40b22d7746f7a10c17924fa2c900f84c30538e097da3147ad67"
Jul 17 19:25:39.448482 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:25:39.448172    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr_odh-ai-gateway-infra(0fbe0be1-629c-4881-a8ff-3598b15b7def)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr" podUID="0fbe0be1-629c-4881-a8ff-3598b15b7def"
Jul 17 19:25:39.472931 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:39.472908    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw"]
Jul 17 19:25:39.611168 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:39.611146    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw"
Jul 17 19:25:39.649600 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:39.649572    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1535dc-ec0a-4743-a2fc-f8848558395c-maas-api-service-ca\") pod \"2f1535dc-ec0a-4743-a2fc-f8848558395c\" (UID: \"2f1535dc-ec0a-4743-a2fc-f8848558395c\") "
Jul 17 19:25:39.649912 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:39.649889    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1535dc-ec0a-4743-a2fc-f8848558395c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2f1535dc-ec0a-4743-a2fc-f8848558395c" (UID: "2f1535dc-ec0a-4743-a2fc-f8848558395c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:25:39.750701 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:39.750626    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1535dc-ec0a-4743-a2fc-f8848558395c-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:25:40.455919 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:40.455830    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw"
Jul 17 19:25:40.455919 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:40.455877    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw" event={"ID":"2f1535dc-ec0a-4743-a2fc-f8848558395c","Type":"ContainerDied","Data":"abe64893f1a76964459d33bcff3dfd6b0e7b8e7f2699345768f25d0232de6870"}
Jul 17 19:25:40.456509 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:40.455935    2559 scope.go:117] "RemoveContainer" containerID="40e7aa9ee06a0d0ae602df48133626ac38facbdf4f507a82452c14bcb2e8f84a"
Jul 17 19:25:40.474908 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:40.474874    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw"]
Jul 17 19:25:40.481739 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:40.481718    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-lj5gw"]
Jul 17 19:25:42.125964 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:42.125913    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f1535dc-ec0a-4743-a2fc-f8848558395c" path="/var/lib/kubelet/pods/2f1535dc-ec0a-4743-a2fc-f8848558395c/volumes"
Jul 17 19:25:51.119640 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:51.119579    2559 scope.go:117] "RemoveContainer" containerID="8a6eec432eafc40b22d7746f7a10c17924fa2c900f84c30538e097da3147ad67"
Jul 17 19:25:51.523028 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:51.522997    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr" event={"ID":"0fbe0be1-629c-4881-a8ff-3598b15b7def","Type":"ContainerStarted","Data":"d1a925bdb5d46be0d0e90c20d646f23acf311c3ac1faf85c6e4f4fb984b804a4"}
Jul 17 19:25:51.541224 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:51.541175    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr" podStartSLOduration=15.54116022 podStartE2EDuration="15.54116022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:25:51.539176353 +0000 UTC m=+3161.980685042" watchObservedRunningTime="2026-07-17 19:25:51.54116022 +0000 UTC m=+3161.982668887"
Jul 17 19:25:52.530909 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:52.530876    2559 generic.go:356] "Generic (PLEG): container finished" podID="0fbe0be1-629c-4881-a8ff-3598b15b7def" containerID="d1a925bdb5d46be0d0e90c20d646f23acf311c3ac1faf85c6e4f4fb984b804a4" exitCode=6
Jul 17 19:25:52.531350 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:52.530949    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr" event={"ID":"0fbe0be1-629c-4881-a8ff-3598b15b7def","Type":"ContainerDied","Data":"d1a925bdb5d46be0d0e90c20d646f23acf311c3ac1faf85c6e4f4fb984b804a4"}
Jul 17 19:25:52.531350 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:52.531011    2559 scope.go:117] "RemoveContainer" containerID="8a6eec432eafc40b22d7746f7a10c17924fa2c900f84c30538e097da3147ad67"
Jul 17 19:25:52.531350 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:52.531301    2559 scope.go:117] "RemoveContainer" containerID="d1a925bdb5d46be0d0e90c20d646f23acf311c3ac1faf85c6e4f4fb984b804a4"
Jul 17 19:25:52.531584 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:25:52.531557    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr_odh-ai-gateway-infra(0fbe0be1-629c-4881-a8ff-3598b15b7def)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr" podUID="0fbe0be1-629c-4881-a8ff-3598b15b7def"
Jul 17 19:25:52.556631 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:52.556608    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr"]
Jul 17 19:25:53.119669 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.119639    2559 scope.go:117] "RemoveContainer" containerID="f41308e3e5641a9084055a3b8d6c0666a7e5bb8dc9f24fdf54374e280a8c1210"
Jul 17 19:25:53.134127 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.134097    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd"]
Jul 17 19:25:53.134593 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.134576    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1535dc-ec0a-4743-a2fc-f8848558395c" containerName="revoke-keys"
Jul 17 19:25:53.134593 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.134594    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1535dc-ec0a-4743-a2fc-f8848558395c" containerName="revoke-keys"
Jul 17 19:25:53.134732 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.134614    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1535dc-ec0a-4743-a2fc-f8848558395c" containerName="revoke-keys"
Jul 17 19:25:53.134732 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.134619    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1535dc-ec0a-4743-a2fc-f8848558395c" containerName="revoke-keys"
Jul 17 19:25:53.134732 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.134682    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f1535dc-ec0a-4743-a2fc-f8848558395c" containerName="revoke-keys"
Jul 17 19:25:53.134732 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.134691    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f1535dc-ec0a-4743-a2fc-f8848558395c" containerName="revoke-keys"
Jul 17 19:25:53.134732 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.134697    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f1535dc-ec0a-4743-a2fc-f8848558395c" containerName="revoke-keys"
Jul 17 19:25:53.138980 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.138962    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd"
Jul 17 19:25:53.144967 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.144932    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd"]
Jul 17 19:25:53.165519 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.165499    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99a188f-a92f-4a18-8565-b7e5687f224c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd\" (UID: \"b99a188f-a92f-4a18-8565-b7e5687f224c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd"
Jul 17 19:25:53.266911 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.266884    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99a188f-a92f-4a18-8565-b7e5687f224c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd\" (UID: \"b99a188f-a92f-4a18-8565-b7e5687f224c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd"
Jul 17 19:25:53.267520 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.267501    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99a188f-a92f-4a18-8565-b7e5687f224c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd\" (UID: \"b99a188f-a92f-4a18-8565-b7e5687f224c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd"
Jul 17 19:25:53.457623 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.457596    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd"
Jul 17 19:25:53.541636 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.541602    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d" event={"ID":"1de6156c-8aca-46a5-b20a-ab378dcc3ecf","Type":"ContainerStarted","Data":"2a9e9ad7fc003602eb2e15137519f19bc5687ab138d708eb1f9d16334042dfd7"}
Jul 17 19:25:53.572525 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.572463    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d" podStartSLOduration=19.572440785 podStartE2EDuration="19.572440785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:25:53.571169277 +0000 UTC m=+3164.012677958" watchObservedRunningTime="2026-07-17 19:25:53.572440785 +0000 UTC m=+3164.013949510"
Jul 17 19:25:53.596228 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.596206    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd"]
Jul 17 19:25:53.597876 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:25:53.597847    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99a188f_a92f_4a18_8565_b7e5687f224c.slice/crio-283f0f9969cfd73f60a93e355086bac68c07aa44fa34e85fb09915bf63e59a91 WatchSource:0}: Error finding container 283f0f9969cfd73f60a93e355086bac68c07aa44fa34e85fb09915bf63e59a91: Status 404 returned error can't find the container with id 283f0f9969cfd73f60a93e355086bac68c07aa44fa34e85fb09915bf63e59a91
Jul 17 19:25:53.675134 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.675111    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr"
Jul 17 19:25:53.771252 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.771157    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fbe0be1-629c-4881-a8ff-3598b15b7def-maas-api-service-ca\") pod \"0fbe0be1-629c-4881-a8ff-3598b15b7def\" (UID: \"0fbe0be1-629c-4881-a8ff-3598b15b7def\") "
Jul 17 19:25:53.771518 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.771497    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fbe0be1-629c-4881-a8ff-3598b15b7def-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0fbe0be1-629c-4881-a8ff-3598b15b7def" (UID: "0fbe0be1-629c-4881-a8ff-3598b15b7def"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:25:53.872426 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:53.872401    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fbe0be1-629c-4881-a8ff-3598b15b7def-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:25:54.461954 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.461866    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-460276-fjvg4"]
Jul 17 19:25:54.462418 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.462396    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fbe0be1-629c-4881-a8ff-3598b15b7def" containerName="revoke-keys"
Jul 17 19:25:54.462553 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.462421    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fbe0be1-629c-4881-a8ff-3598b15b7def" containerName="revoke-keys"
Jul 17 19:25:54.462553 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.462438    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fbe0be1-629c-4881-a8ff-3598b15b7def" containerName="revoke-keys"
Jul 17 19:25:54.462553 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.462446    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fbe0be1-629c-4881-a8ff-3598b15b7def" containerName="revoke-keys"
Jul 17 19:25:54.462553 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.462456    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1535dc-ec0a-4743-a2fc-f8848558395c" containerName="revoke-keys"
Jul 17 19:25:54.462553 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.462465    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1535dc-ec0a-4743-a2fc-f8848558395c" containerName="revoke-keys"
Jul 17 19:25:54.462553 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.462484    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fbe0be1-629c-4881-a8ff-3598b15b7def" containerName="revoke-keys"
Jul 17 19:25:54.462553 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.462492    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fbe0be1-629c-4881-a8ff-3598b15b7def" containerName="revoke-keys"
Jul 17 19:25:54.462890 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.462608    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fbe0be1-629c-4881-a8ff-3598b15b7def" containerName="revoke-keys"
Jul 17 19:25:54.462890 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.462622    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fbe0be1-629c-4881-a8ff-3598b15b7def" containerName="revoke-keys"
Jul 17 19:25:54.462890 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.462631    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fbe0be1-629c-4881-a8ff-3598b15b7def" containerName="revoke-keys"
Jul 17 19:25:54.465799 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.465771    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-460276-fjvg4"
Jul 17 19:25:54.474050 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.473939    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-460276-fjvg4"]
Jul 17 19:25:54.548997 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.548958    2559 generic.go:356] "Generic (PLEG): container finished" podID="b99a188f-a92f-4a18-8565-b7e5687f224c" containerID="44a4e98916202692c8d6df64ae9b895eff76c1d7925650b872df0502ebdb253d" exitCode=6
Jul 17 19:25:54.549481 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.549053    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd" event={"ID":"b99a188f-a92f-4a18-8565-b7e5687f224c","Type":"ContainerDied","Data":"44a4e98916202692c8d6df64ae9b895eff76c1d7925650b872df0502ebdb253d"}
Jul 17 19:25:54.549481 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.549113    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd" event={"ID":"b99a188f-a92f-4a18-8565-b7e5687f224c","Type":"ContainerStarted","Data":"283f0f9969cfd73f60a93e355086bac68c07aa44fa34e85fb09915bf63e59a91"}
Jul 17 19:25:54.549481 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.549232    2559 scope.go:117] "RemoveContainer" containerID="44a4e98916202692c8d6df64ae9b895eff76c1d7925650b872df0502ebdb253d"
Jul 17 19:25:54.550648 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.550631    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr"
Jul 17 19:25:54.550648 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.550645    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr" event={"ID":"0fbe0be1-629c-4881-a8ff-3598b15b7def","Type":"ContainerDied","Data":"d49ae14fd71343b7add1fd70413aa6009b5b29d89de35b2d9a4a28c178fe5ddc"}
Jul 17 19:25:54.550806 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.550691    2559 scope.go:117] "RemoveContainer" containerID="d1a925bdb5d46be0d0e90c20d646f23acf311c3ac1faf85c6e4f4fb984b804a4"
Jul 17 19:25:54.553360 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.553342    2559 generic.go:356] "Generic (PLEG): container finished" podID="1de6156c-8aca-46a5-b20a-ab378dcc3ecf" containerID="2a9e9ad7fc003602eb2e15137519f19bc5687ab138d708eb1f9d16334042dfd7" exitCode=6
Jul 17 19:25:54.553468 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.553391    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d" event={"ID":"1de6156c-8aca-46a5-b20a-ab378dcc3ecf","Type":"ContainerDied","Data":"2a9e9ad7fc003602eb2e15137519f19bc5687ab138d708eb1f9d16334042dfd7"}
Jul 17 19:25:54.553756 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.553736    2559 scope.go:117] "RemoveContainer" containerID="2a9e9ad7fc003602eb2e15137519f19bc5687ab138d708eb1f9d16334042dfd7"
Jul 17 19:25:54.553942 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:25:54.553922    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d_odh-ai-gateway-infra(1de6156c-8aca-46a5-b20a-ab378dcc3ecf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d" podUID="1de6156c-8aca-46a5-b20a-ab378dcc3ecf"
Jul 17 19:25:54.562485 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.562456    2559 scope.go:117] "RemoveContainer" containerID="f41308e3e5641a9084055a3b8d6c0666a7e5bb8dc9f24fdf54374e280a8c1210"
Jul 17 19:25:54.578889 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.578865    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94f553d-0c14-415b-97a8-ac1ec371f547-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-460276-fjvg4\" (UID: \"c94f553d-0c14-415b-97a8-ac1ec371f547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-460276-fjvg4"
Jul 17 19:25:54.581455 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.581437    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr"]
Jul 17 19:25:54.585059 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.585035    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d"]
Jul 17 19:25:54.588387 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.588368    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-pflpr"]
Jul 17 19:25:54.680469 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.680443    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94f553d-0c14-415b-97a8-ac1ec371f547-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-460276-fjvg4\" (UID: \"c94f553d-0c14-415b-97a8-ac1ec371f547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-460276-fjvg4"
Jul 17 19:25:54.681030 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.681010    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94f553d-0c14-415b-97a8-ac1ec371f547-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-460276-fjvg4\" (UID: \"c94f553d-0c14-415b-97a8-ac1ec371f547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-460276-fjvg4"
Jul 17 19:25:54.779382 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.779360    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-460276-fjvg4"
Jul 17 19:25:54.921445 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:54.921419    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-460276-fjvg4"]
Jul 17 19:25:54.923363 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:25:54.923301    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94f553d_0c14_415b_97a8_ac1ec371f547.slice/crio-a42391916a0772330b504183094cc808e1643c9fca3328072de03c31d9cf25a0 WatchSource:0}: Error finding container a42391916a0772330b504183094cc808e1643c9fca3328072de03c31d9cf25a0: Status 404 returned error can't find the container with id a42391916a0772330b504183094cc808e1643c9fca3328072de03c31d9cf25a0
Jul 17 19:25:55.560768 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:55.560726    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-460276-fjvg4" event={"ID":"c94f553d-0c14-415b-97a8-ac1ec371f547","Type":"ContainerStarted","Data":"7fe43c8e684f5873b32c2b366dea94a8c5b05a68b209195be65545d023316c9f"}
Jul 17 19:25:55.561238 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:55.560788    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-460276-fjvg4" event={"ID":"c94f553d-0c14-415b-97a8-ac1ec371f547","Type":"ContainerStarted","Data":"a42391916a0772330b504183094cc808e1643c9fca3328072de03c31d9cf25a0"}
Jul 17 19:25:55.565600 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:55.565564    2559 generic.go:356] "Generic (PLEG): container finished" podID="b99a188f-a92f-4a18-8565-b7e5687f224c" containerID="b8ef9756f313bf7bd5856da645401c0e0bc256542396032c35995ee7ec882a8a" exitCode=6
Jul 17 19:25:55.565738 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:55.565628    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd" event={"ID":"b99a188f-a92f-4a18-8565-b7e5687f224c","Type":"ContainerDied","Data":"b8ef9756f313bf7bd5856da645401c0e0bc256542396032c35995ee7ec882a8a"}
Jul 17 19:25:55.565738 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:55.565684    2559 scope.go:117] "RemoveContainer" containerID="44a4e98916202692c8d6df64ae9b895eff76c1d7925650b872df0502ebdb253d"
Jul 17 19:25:55.565866 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:55.565848    2559 scope.go:117] "RemoveContainer" containerID="b8ef9756f313bf7bd5856da645401c0e0bc256542396032c35995ee7ec882a8a"
Jul 17 19:25:55.566086 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:25:55.566061    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd_odh-ai-gateway-infra(b99a188f-a92f-4a18-8565-b7e5687f224c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd" podUID="b99a188f-a92f-4a18-8565-b7e5687f224c"
Jul 17 19:25:55.579274 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:55.579228    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-460276-fjvg4" podStartSLOduration=1.5792102529999998 podStartE2EDuration="1.579210253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:25:55.575638074 +0000 UTC m=+3166.017146765" watchObservedRunningTime="2026-07-17 19:25:55.579210253 +0000 UTC m=+3166.020718944"
Jul 17 19:25:55.712417 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:55.712395    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d"
Jul 17 19:25:55.791026 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:55.790991    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de6156c-8aca-46a5-b20a-ab378dcc3ecf-maas-api-service-ca\") pod \"1de6156c-8aca-46a5-b20a-ab378dcc3ecf\" (UID: \"1de6156c-8aca-46a5-b20a-ab378dcc3ecf\") "
Jul 17 19:25:55.791366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:55.791344    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1de6156c-8aca-46a5-b20a-ab378dcc3ecf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1de6156c-8aca-46a5-b20a-ab378dcc3ecf" (UID: "1de6156c-8aca-46a5-b20a-ab378dcc3ecf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:25:55.892223 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:55.892149    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de6156c-8aca-46a5-b20a-ab378dcc3ecf-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:25:56.126701 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:56.126668    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fbe0be1-629c-4881-a8ff-3598b15b7def" path="/var/lib/kubelet/pods/0fbe0be1-629c-4881-a8ff-3598b15b7def/volumes"
Jul 17 19:25:56.574379 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:56.574350    2559 generic.go:356] "Generic (PLEG): container finished" podID="c94f553d-0c14-415b-97a8-ac1ec371f547" containerID="7fe43c8e684f5873b32c2b366dea94a8c5b05a68b209195be65545d023316c9f" exitCode=0
Jul 17 19:25:56.574791 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:56.574431    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-460276-fjvg4" event={"ID":"c94f553d-0c14-415b-97a8-ac1ec371f547","Type":"ContainerDied","Data":"7fe43c8e684f5873b32c2b366dea94a8c5b05a68b209195be65545d023316c9f"}
Jul 17 19:25:56.575775 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:56.575752    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d"
Jul 17 19:25:56.575918 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:56.575755    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d" event={"ID":"1de6156c-8aca-46a5-b20a-ab378dcc3ecf","Type":"ContainerDied","Data":"cdb2ddb96ad026cef076ca4cc43ac74078f7524a95a5ae6f20451e5ba7c01646"}
Jul 17 19:25:56.575918 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:56.575893    2559 scope.go:117] "RemoveContainer" containerID="2a9e9ad7fc003602eb2e15137519f19bc5687ab138d708eb1f9d16334042dfd7"
Jul 17 19:25:56.577747 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:56.577730    2559 scope.go:117] "RemoveContainer" containerID="b8ef9756f313bf7bd5856da645401c0e0bc256542396032c35995ee7ec882a8a"
Jul 17 19:25:56.577969 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:25:56.577945    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd_odh-ai-gateway-infra(b99a188f-a92f-4a18-8565-b7e5687f224c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd" podUID="b99a188f-a92f-4a18-8565-b7e5687f224c"
Jul 17 19:25:56.603848 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:56.603825    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d"]
Jul 17 19:25:56.611131 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:56.611102    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-k7k6d"]
Jul 17 19:25:57.714328 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:57.714283    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-460276-fjvg4"
Jul 17 19:25:57.810458 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:57.810430    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94f553d-0c14-415b-97a8-ac1ec371f547-maas-api-service-ca\") pod \"c94f553d-0c14-415b-97a8-ac1ec371f547\" (UID: \"c94f553d-0c14-415b-97a8-ac1ec371f547\") "
Jul 17 19:25:57.810778 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:57.810755    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94f553d-0c14-415b-97a8-ac1ec371f547-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c94f553d-0c14-415b-97a8-ac1ec371f547" (UID: "c94f553d-0c14-415b-97a8-ac1ec371f547"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:25:57.911610 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:57.911539    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94f553d-0c14-415b-97a8-ac1ec371f547-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:25:58.126623 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:58.126590    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1de6156c-8aca-46a5-b20a-ab378dcc3ecf" path="/var/lib/kubelet/pods/1de6156c-8aca-46a5-b20a-ab378dcc3ecf/volumes"
Jul 17 19:25:58.591255 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:58.591227    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-460276-fjvg4"
Jul 17 19:25:58.591470 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:58.591264    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-460276-fjvg4" event={"ID":"c94f553d-0c14-415b-97a8-ac1ec371f547","Type":"ContainerDied","Data":"a42391916a0772330b504183094cc808e1643c9fca3328072de03c31d9cf25a0"}
Jul 17 19:25:58.591470 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:25:58.591326    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a42391916a0772330b504183094cc808e1643c9fca3328072de03c31d9cf25a0"
Jul 17 19:26:09.120315 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:09.120276    2559 scope.go:117] "RemoveContainer" containerID="b8ef9756f313bf7bd5856da645401c0e0bc256542396032c35995ee7ec882a8a"
Jul 17 19:26:09.656825 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:09.656789    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd" event={"ID":"b99a188f-a92f-4a18-8565-b7e5687f224c","Type":"ContainerStarted","Data":"962236ad5887c08e7884a568e1b80f99aca4e042f7c02a9d560a2aca87ee50c2"}
Jul 17 19:26:09.676259 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:09.676213    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd" podStartSLOduration=16.676198882 podStartE2EDuration="16.676198882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:26:09.674252607 +0000 UTC m=+3180.115761296" watchObservedRunningTime="2026-07-17 19:26:09.676198882 +0000 UTC m=+3180.117707571"
Jul 17 19:26:10.664029 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:10.664000    2559 generic.go:356] "Generic (PLEG): container finished" podID="b99a188f-a92f-4a18-8565-b7e5687f224c" containerID="962236ad5887c08e7884a568e1b80f99aca4e042f7c02a9d560a2aca87ee50c2" exitCode=6
Jul 17 19:26:10.664414 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:10.664064    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd" event={"ID":"b99a188f-a92f-4a18-8565-b7e5687f224c","Type":"ContainerDied","Data":"962236ad5887c08e7884a568e1b80f99aca4e042f7c02a9d560a2aca87ee50c2"}
Jul 17 19:26:10.664414 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:10.664123    2559 scope.go:117] "RemoveContainer" containerID="b8ef9756f313bf7bd5856da645401c0e0bc256542396032c35995ee7ec882a8a"
Jul 17 19:26:10.664414 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:10.664379    2559 scope.go:117] "RemoveContainer" containerID="962236ad5887c08e7884a568e1b80f99aca4e042f7c02a9d560a2aca87ee50c2"
Jul 17 19:26:10.664622 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:26:10.664600    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd_odh-ai-gateway-infra(b99a188f-a92f-4a18-8565-b7e5687f224c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd" podUID="b99a188f-a92f-4a18-8565-b7e5687f224c"
Jul 17 19:26:10.687318 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:10.687281    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd"]
Jul 17 19:26:11.405364 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.405328    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl"]
Jul 17 19:26:11.405730 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.405718    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c94f553d-0c14-415b-97a8-ac1ec371f547" containerName="revoke-keys"
Jul 17 19:26:11.405788 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.405732    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c94f553d-0c14-415b-97a8-ac1ec371f547" containerName="revoke-keys"
Jul 17 19:26:11.405788 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.405759    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de6156c-8aca-46a5-b20a-ab378dcc3ecf" containerName="revoke-keys"
Jul 17 19:26:11.405788 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.405764    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de6156c-8aca-46a5-b20a-ab378dcc3ecf" containerName="revoke-keys"
Jul 17 19:26:11.405788 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.405771    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de6156c-8aca-46a5-b20a-ab378dcc3ecf" containerName="revoke-keys"
Jul 17 19:26:11.405788 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.405776    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de6156c-8aca-46a5-b20a-ab378dcc3ecf" containerName="revoke-keys"
Jul 17 19:26:11.405788 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.405784    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de6156c-8aca-46a5-b20a-ab378dcc3ecf" containerName="revoke-keys"
Jul 17 19:26:11.405788 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.405790    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de6156c-8aca-46a5-b20a-ab378dcc3ecf" containerName="revoke-keys"
Jul 17 19:26:11.406032 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.405861    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de6156c-8aca-46a5-b20a-ab378dcc3ecf" containerName="revoke-keys"
Jul 17 19:26:11.406032 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.405869    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de6156c-8aca-46a5-b20a-ab378dcc3ecf" containerName="revoke-keys"
Jul 17 19:26:11.406032 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.405878    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c94f553d-0c14-415b-97a8-ac1ec371f547" containerName="revoke-keys"
Jul 17 19:26:11.413876 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.412061    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl"
Jul 17 19:26:11.417097 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.417068    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl"]
Jul 17 19:26:11.538688 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.538647    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a070c252-0c42-42fb-8d01-94d0fe07cfe6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-26jwl\" (UID: \"a070c252-0c42-42fb-8d01-94d0fe07cfe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl"
Jul 17 19:26:11.639931 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.639897    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a070c252-0c42-42fb-8d01-94d0fe07cfe6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-26jwl\" (UID: \"a070c252-0c42-42fb-8d01-94d0fe07cfe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl"
Jul 17 19:26:11.640568 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.640547    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a070c252-0c42-42fb-8d01-94d0fe07cfe6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-26jwl\" (UID: \"a070c252-0c42-42fb-8d01-94d0fe07cfe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl"
Jul 17 19:26:11.724746 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.724713    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl"
Jul 17 19:26:11.817669 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.817641    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd"
Jul 17 19:26:11.872933 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.872900    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl"]
Jul 17 19:26:11.874811 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:26:11.874780    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda070c252_0c42_42fb_8d01_94d0fe07cfe6.slice/crio-3ebdae62e37954b4574a1f20d7709e2e372d949da6f442c5f84c41f3fcfc7c6a WatchSource:0}: Error finding container 3ebdae62e37954b4574a1f20d7709e2e372d949da6f442c5f84c41f3fcfc7c6a: Status 404 returned error can't find the container with id 3ebdae62e37954b4574a1f20d7709e2e372d949da6f442c5f84c41f3fcfc7c6a
Jul 17 19:26:11.943425 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.943362    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99a188f-a92f-4a18-8565-b7e5687f224c-maas-api-service-ca\") pod \"b99a188f-a92f-4a18-8565-b7e5687f224c\" (UID: \"b99a188f-a92f-4a18-8565-b7e5687f224c\") "
Jul 17 19:26:11.943740 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:11.943720    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b99a188f-a92f-4a18-8565-b7e5687f224c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b99a188f-a92f-4a18-8565-b7e5687f224c" (UID: "b99a188f-a92f-4a18-8565-b7e5687f224c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:26:12.044515 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:12.044490    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99a188f-a92f-4a18-8565-b7e5687f224c-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:26:12.678873 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:12.678844    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd"
Jul 17 19:26:12.679077 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:12.678885    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd" event={"ID":"b99a188f-a92f-4a18-8565-b7e5687f224c","Type":"ContainerDied","Data":"283f0f9969cfd73f60a93e355086bac68c07aa44fa34e85fb09915bf63e59a91"}
Jul 17 19:26:12.679077 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:12.678958    2559 scope.go:117] "RemoveContainer" containerID="962236ad5887c08e7884a568e1b80f99aca4e042f7c02a9d560a2aca87ee50c2"
Jul 17 19:26:12.681967 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:12.681941    2559 generic.go:356] "Generic (PLEG): container finished" podID="a070c252-0c42-42fb-8d01-94d0fe07cfe6" containerID="5f346763a4a490dd1a98254e6575cfeb7089c3a2d923c28a21bfd78a6ec936c7" exitCode=6
Jul 17 19:26:12.682105 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:12.682018    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl" event={"ID":"a070c252-0c42-42fb-8d01-94d0fe07cfe6","Type":"ContainerDied","Data":"5f346763a4a490dd1a98254e6575cfeb7089c3a2d923c28a21bfd78a6ec936c7"}
Jul 17 19:26:12.682105 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:12.682057    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl" event={"ID":"a070c252-0c42-42fb-8d01-94d0fe07cfe6","Type":"ContainerStarted","Data":"3ebdae62e37954b4574a1f20d7709e2e372d949da6f442c5f84c41f3fcfc7c6a"}
Jul 17 19:26:12.682217 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:12.682152    2559 scope.go:117] "RemoveContainer" containerID="5f346763a4a490dd1a98254e6575cfeb7089c3a2d923c28a21bfd78a6ec936c7"
Jul 17 19:26:12.696197 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:12.696175    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd"]
Jul 17 19:26:12.699926 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:12.699904    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-x8zhd"]
Jul 17 19:26:13.689858 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:13.689826    2559 generic.go:356] "Generic (PLEG): container finished" podID="a070c252-0c42-42fb-8d01-94d0fe07cfe6" containerID="38e7f190a564a3cac30edd3feb339e5ae72474b04b44fb455e6d1b342d11c7fc" exitCode=6
Jul 17 19:26:13.690335 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:13.689908    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl" event={"ID":"a070c252-0c42-42fb-8d01-94d0fe07cfe6","Type":"ContainerDied","Data":"38e7f190a564a3cac30edd3feb339e5ae72474b04b44fb455e6d1b342d11c7fc"}
Jul 17 19:26:13.690335 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:13.689974    2559 scope.go:117] "RemoveContainer" containerID="5f346763a4a490dd1a98254e6575cfeb7089c3a2d923c28a21bfd78a6ec936c7"
Jul 17 19:26:13.690335 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:13.690086    2559 scope.go:117] "RemoveContainer" containerID="38e7f190a564a3cac30edd3feb339e5ae72474b04b44fb455e6d1b342d11c7fc"
Jul 17 19:26:13.690505 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:26:13.690444    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-26jwl_odh-ai-gateway-infra(a070c252-0c42-42fb-8d01-94d0fe07cfe6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl" podUID="a070c252-0c42-42fb-8d01-94d0fe07cfe6"
Jul 17 19:26:14.126435 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:14.126400    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b99a188f-a92f-4a18-8565-b7e5687f224c" path="/var/lib/kubelet/pods/b99a188f-a92f-4a18-8565-b7e5687f224c/volumes"
Jul 17 19:26:14.699871 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:14.699846    2559 scope.go:117] "RemoveContainer" containerID="38e7f190a564a3cac30edd3feb339e5ae72474b04b44fb455e6d1b342d11c7fc"
Jul 17 19:26:14.700283 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:26:14.700052    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-26jwl_odh-ai-gateway-infra(a070c252-0c42-42fb-8d01-94d0fe07cfe6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl" podUID="a070c252-0c42-42fb-8d01-94d0fe07cfe6"
Jul 17 19:26:25.119876 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:25.119843    2559 scope.go:117] "RemoveContainer" containerID="38e7f190a564a3cac30edd3feb339e5ae72474b04b44fb455e6d1b342d11c7fc"
Jul 17 19:26:25.766379 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:25.766280    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl" event={"ID":"a070c252-0c42-42fb-8d01-94d0fe07cfe6","Type":"ContainerStarted","Data":"4b70934931efa9fe25d3e1fcaf084e016115e966ed781f26d494c071f3145032"}
Jul 17 19:26:25.783179 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:25.783126    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl" podStartSLOduration=14.783111185 podStartE2EDuration="14.783111185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:26:25.781562928 +0000 UTC m=+3196.223071619" watchObservedRunningTime="2026-07-17 19:26:25.783111185 +0000 UTC m=+3196.224619950"
Jul 17 19:26:26.774672 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:26.774639    2559 generic.go:356] "Generic (PLEG): container finished" podID="a070c252-0c42-42fb-8d01-94d0fe07cfe6" containerID="4b70934931efa9fe25d3e1fcaf084e016115e966ed781f26d494c071f3145032" exitCode=6
Jul 17 19:26:26.775076 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:26.774710    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl" event={"ID":"a070c252-0c42-42fb-8d01-94d0fe07cfe6","Type":"ContainerDied","Data":"4b70934931efa9fe25d3e1fcaf084e016115e966ed781f26d494c071f3145032"}
Jul 17 19:26:26.775076 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:26.774778    2559 scope.go:117] "RemoveContainer" containerID="38e7f190a564a3cac30edd3feb339e5ae72474b04b44fb455e6d1b342d11c7fc"
Jul 17 19:26:26.775076 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:26.775053    2559 scope.go:117] "RemoveContainer" containerID="4b70934931efa9fe25d3e1fcaf084e016115e966ed781f26d494c071f3145032"
Jul 17 19:26:26.775285 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:26:26.775262    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-26jwl_odh-ai-gateway-infra(a070c252-0c42-42fb-8d01-94d0fe07cfe6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl" podUID="a070c252-0c42-42fb-8d01-94d0fe07cfe6"
Jul 17 19:26:26.802513 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:26.802482    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl"]
Jul 17 19:26:27.916229 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:27.916201    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl"
Jul 17 19:26:27.985122 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:27.985100    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a070c252-0c42-42fb-8d01-94d0fe07cfe6-maas-api-service-ca\") pod \"a070c252-0c42-42fb-8d01-94d0fe07cfe6\" (UID: \"a070c252-0c42-42fb-8d01-94d0fe07cfe6\") "
Jul 17 19:26:27.985457 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:27.985439    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a070c252-0c42-42fb-8d01-94d0fe07cfe6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a070c252-0c42-42fb-8d01-94d0fe07cfe6" (UID: "a070c252-0c42-42fb-8d01-94d0fe07cfe6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:26:28.085921 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:28.085887    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a070c252-0c42-42fb-8d01-94d0fe07cfe6-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:26:28.789052 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:28.789024    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl"
Jul 17 19:26:28.789287 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:28.789066    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl" event={"ID":"a070c252-0c42-42fb-8d01-94d0fe07cfe6","Type":"ContainerDied","Data":"3ebdae62e37954b4574a1f20d7709e2e372d949da6f442c5f84c41f3fcfc7c6a"}
Jul 17 19:26:28.789287 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:28.789132    2559 scope.go:117] "RemoveContainer" containerID="4b70934931efa9fe25d3e1fcaf084e016115e966ed781f26d494c071f3145032"
Jul 17 19:26:28.811000 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:28.810973    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl"]
Jul 17 19:26:28.815544 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:28.815516    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-26jwl"]
Jul 17 19:26:30.126359 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:30.126324    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a070c252-0c42-42fb-8d01-94d0fe07cfe6" path="/var/lib/kubelet/pods/a070c252-0c42-42fb-8d01-94d0fe07cfe6/volumes"
Jul 17 19:26:34.293524 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.293492    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc"]
Jul 17 19:26:34.293926 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.293855    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b99a188f-a92f-4a18-8565-b7e5687f224c" containerName="revoke-keys"
Jul 17 19:26:34.293926 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.293865    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b99a188f-a92f-4a18-8565-b7e5687f224c" containerName="revoke-keys"
Jul 17 19:26:34.293926 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.293877    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a070c252-0c42-42fb-8d01-94d0fe07cfe6" containerName="revoke-keys"
Jul 17 19:26:34.293926 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.293882    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a070c252-0c42-42fb-8d01-94d0fe07cfe6" containerName="revoke-keys"
Jul 17 19:26:34.293926 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.293895    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a070c252-0c42-42fb-8d01-94d0fe07cfe6" containerName="revoke-keys"
Jul 17 19:26:34.293926 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.293901    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a070c252-0c42-42fb-8d01-94d0fe07cfe6" containerName="revoke-keys"
Jul 17 19:26:34.293926 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.293916    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b99a188f-a92f-4a18-8565-b7e5687f224c" containerName="revoke-keys"
Jul 17 19:26:34.293926 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.293921    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b99a188f-a92f-4a18-8565-b7e5687f224c" containerName="revoke-keys"
Jul 17 19:26:34.294171 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.293979    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de6156c-8aca-46a5-b20a-ab378dcc3ecf" containerName="revoke-keys"
Jul 17 19:26:34.294171 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.293988    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b99a188f-a92f-4a18-8565-b7e5687f224c" containerName="revoke-keys"
Jul 17 19:26:34.294171 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.293994    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a070c252-0c42-42fb-8d01-94d0fe07cfe6" containerName="revoke-keys"
Jul 17 19:26:34.294171 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.293999    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b99a188f-a92f-4a18-8565-b7e5687f224c" containerName="revoke-keys"
Jul 17 19:26:34.294171 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.294004    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a070c252-0c42-42fb-8d01-94d0fe07cfe6" containerName="revoke-keys"
Jul 17 19:26:34.294171 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.294012    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a070c252-0c42-42fb-8d01-94d0fe07cfe6" containerName="revoke-keys"
Jul 17 19:26:34.298685 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.298669    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc"
Jul 17 19:26:34.301852 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.301826    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:26:34.304521 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.304497    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc"]
Jul 17 19:26:34.334875 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.334843    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/441bc1e2-a123-4824-a47b-8d52b7fed562-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc\" (UID: \"441bc1e2-a123-4824-a47b-8d52b7fed562\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc"
Jul 17 19:26:34.435941 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.435910    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/441bc1e2-a123-4824-a47b-8d52b7fed562-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc\" (UID: \"441bc1e2-a123-4824-a47b-8d52b7fed562\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc"
Jul 17 19:26:34.436533 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.436499    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/441bc1e2-a123-4824-a47b-8d52b7fed562-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc\" (UID: \"441bc1e2-a123-4824-a47b-8d52b7fed562\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc"
Jul 17 19:26:34.611010 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.610968    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc"
Jul 17 19:26:34.769055 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.769030    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc"]
Jul 17 19:26:34.770485 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:26:34.770447    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441bc1e2_a123_4824_a47b_8d52b7fed562.slice/crio-efb3e7bc3dec952e74acc609ca536a65e05f6abc8dcd99ccad1ee53355b0848a WatchSource:0}: Error finding container efb3e7bc3dec952e74acc609ca536a65e05f6abc8dcd99ccad1ee53355b0848a: Status 404 returned error can't find the container with id efb3e7bc3dec952e74acc609ca536a65e05f6abc8dcd99ccad1ee53355b0848a
Jul 17 19:26:34.827905 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:34.827876    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc" event={"ID":"441bc1e2-a123-4824-a47b-8d52b7fed562","Type":"ContainerStarted","Data":"efb3e7bc3dec952e74acc609ca536a65e05f6abc8dcd99ccad1ee53355b0848a"}
Jul 17 19:26:35.835888 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:35.835854    2559 generic.go:356] "Generic (PLEG): container finished" podID="441bc1e2-a123-4824-a47b-8d52b7fed562" containerID="caf7072f636c87544bbd93f7466c853e7c75c8308e1dc341fe9aa9624a9c868e" exitCode=6
Jul 17 19:26:35.836294 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:35.835941    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc" event={"ID":"441bc1e2-a123-4824-a47b-8d52b7fed562","Type":"ContainerDied","Data":"caf7072f636c87544bbd93f7466c853e7c75c8308e1dc341fe9aa9624a9c868e"}
Jul 17 19:26:35.836294 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:35.836135    2559 scope.go:117] "RemoveContainer" containerID="caf7072f636c87544bbd93f7466c853e7c75c8308e1dc341fe9aa9624a9c868e"
Jul 17 19:26:36.843594 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:36.843556    2559 generic.go:356] "Generic (PLEG): container finished" podID="441bc1e2-a123-4824-a47b-8d52b7fed562" containerID="7f5b2c8c2b62b69a77bd92967001e2de9111770f936ada9782a1c3415f12fbaa" exitCode=6
Jul 17 19:26:36.844015 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:36.843643    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc" event={"ID":"441bc1e2-a123-4824-a47b-8d52b7fed562","Type":"ContainerDied","Data":"7f5b2c8c2b62b69a77bd92967001e2de9111770f936ada9782a1c3415f12fbaa"}
Jul 17 19:26:36.844015 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:36.843698    2559 scope.go:117] "RemoveContainer" containerID="caf7072f636c87544bbd93f7466c853e7c75c8308e1dc341fe9aa9624a9c868e"
Jul 17 19:26:36.844015 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:36.843806    2559 scope.go:117] "RemoveContainer" containerID="7f5b2c8c2b62b69a77bd92967001e2de9111770f936ada9782a1c3415f12fbaa"
Jul 17 19:26:36.844131 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:26:36.844075    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc_odh-ai-gateway-infra(441bc1e2-a123-4824-a47b-8d52b7fed562)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc" podUID="441bc1e2-a123-4824-a47b-8d52b7fed562"
Jul 17 19:26:37.851641 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:37.851614    2559 scope.go:117] "RemoveContainer" containerID="7f5b2c8c2b62b69a77bd92967001e2de9111770f936ada9782a1c3415f12fbaa"
Jul 17 19:26:37.852031 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:26:37.851843    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc_odh-ai-gateway-infra(441bc1e2-a123-4824-a47b-8d52b7fed562)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc" podUID="441bc1e2-a123-4824-a47b-8d52b7fed562"
Jul 17 19:26:43.179148 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:43.179109    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf"]
Jul 17 19:26:43.180009 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:43.179972    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a070c252-0c42-42fb-8d01-94d0fe07cfe6" containerName="revoke-keys"
Jul 17 19:26:43.180009 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:43.179998    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a070c252-0c42-42fb-8d01-94d0fe07cfe6" containerName="revoke-keys"
Jul 17 19:26:43.180263 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:43.180073    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b99a188f-a92f-4a18-8565-b7e5687f224c" containerName="revoke-keys"
Jul 17 19:26:43.180263 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:43.180092    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b99a188f-a92f-4a18-8565-b7e5687f224c" containerName="revoke-keys"
Jul 17 19:26:43.180402 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:43.180338    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b99a188f-a92f-4a18-8565-b7e5687f224c" containerName="revoke-keys"
Jul 17 19:26:43.185768 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:43.185748    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf"
Jul 17 19:26:43.189608 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:43.189574    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf"]
Jul 17 19:26:43.314787 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:43.314753    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/014e23b1-4e20-4bba-9048-d998ed6e52b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf\" (UID: \"014e23b1-4e20-4bba-9048-d998ed6e52b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf"
Jul 17 19:26:43.415377 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:43.415345    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/014e23b1-4e20-4bba-9048-d998ed6e52b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf\" (UID: \"014e23b1-4e20-4bba-9048-d998ed6e52b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf"
Jul 17 19:26:43.415930 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:43.415904    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/014e23b1-4e20-4bba-9048-d998ed6e52b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf\" (UID: \"014e23b1-4e20-4bba-9048-d998ed6e52b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf"
Jul 17 19:26:43.498707 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:43.498653    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf"
Jul 17 19:26:43.628950 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:43.628927    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf"]
Jul 17 19:26:43.631152 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:26:43.631122    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014e23b1_4e20_4bba_9048_d998ed6e52b3.slice/crio-6a4cea8f8e2e30bf8b7888a8edf210f9705157f5040ae4af90a26ceb311fd44b WatchSource:0}: Error finding container 6a4cea8f8e2e30bf8b7888a8edf210f9705157f5040ae4af90a26ceb311fd44b: Status 404 returned error can't find the container with id 6a4cea8f8e2e30bf8b7888a8edf210f9705157f5040ae4af90a26ceb311fd44b
Jul 17 19:26:43.887141 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:43.887104    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf" event={"ID":"014e23b1-4e20-4bba-9048-d998ed6e52b3","Type":"ContainerStarted","Data":"6a4cea8f8e2e30bf8b7888a8edf210f9705157f5040ae4af90a26ceb311fd44b"}
Jul 17 19:26:44.894862 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:44.894827    2559 generic.go:356] "Generic (PLEG): container finished" podID="014e23b1-4e20-4bba-9048-d998ed6e52b3" containerID="a452dbe8e7d8cb6af6f0f4d8e12e56d1e60ae0734afff802810283ca2a17d386" exitCode=6
Jul 17 19:26:44.895374 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:44.894869    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf" event={"ID":"014e23b1-4e20-4bba-9048-d998ed6e52b3","Type":"ContainerDied","Data":"a452dbe8e7d8cb6af6f0f4d8e12e56d1e60ae0734afff802810283ca2a17d386"}
Jul 17 19:26:44.895374 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:44.895126    2559 scope.go:117] "RemoveContainer" containerID="a452dbe8e7d8cb6af6f0f4d8e12e56d1e60ae0734afff802810283ca2a17d386"
Jul 17 19:26:45.903517 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:45.903481    2559 generic.go:356] "Generic (PLEG): container finished" podID="014e23b1-4e20-4bba-9048-d998ed6e52b3" containerID="f8ce99635915d2d1bf76098b938c089de33de97e08e3cb6e10d9fc7ba3a08356" exitCode=6
Jul 17 19:26:45.903925 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:45.903547    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf" event={"ID":"014e23b1-4e20-4bba-9048-d998ed6e52b3","Type":"ContainerDied","Data":"f8ce99635915d2d1bf76098b938c089de33de97e08e3cb6e10d9fc7ba3a08356"}
Jul 17 19:26:45.903925 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:45.903609    2559 scope.go:117] "RemoveContainer" containerID="a452dbe8e7d8cb6af6f0f4d8e12e56d1e60ae0734afff802810283ca2a17d386"
Jul 17 19:26:45.903925 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:45.903828    2559 scope.go:117] "RemoveContainer" containerID="f8ce99635915d2d1bf76098b938c089de33de97e08e3cb6e10d9fc7ba3a08356"
Jul 17 19:26:45.904108 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:26:45.904085    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf_odh-ai-gateway-infra(014e23b1-4e20-4bba-9048-d998ed6e52b3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf" podUID="014e23b1-4e20-4bba-9048-d998ed6e52b3"
Jul 17 19:26:46.911193 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:46.911166    2559 scope.go:117] "RemoveContainer" containerID="f8ce99635915d2d1bf76098b938c089de33de97e08e3cb6e10d9fc7ba3a08356"
Jul 17 19:26:46.911636 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:26:46.911365    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf_odh-ai-gateway-infra(014e23b1-4e20-4bba-9048-d998ed6e52b3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf" podUID="014e23b1-4e20-4bba-9048-d998ed6e52b3"
Jul 17 19:26:49.801844 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.801807    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"]
Jul 17 19:26:49.805504 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.805485    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:26:49.809060 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.809017    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-98e305\"/\"auth-test-model-98e305-kserve-self-signed-certs\""
Jul 17 19:26:49.809360 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.809339    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-98e305\"/\"kube-root-ca.crt\""
Jul 17 19:26:49.809990 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.809806    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-98e305\"/\"openshift-service-ca.crt\""
Jul 17 19:26:49.809990 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.809905    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-98e305\"/\"default-dockercfg-5qjtz\""
Jul 17 19:26:49.813765 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.813277    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"]
Jul 17 19:26:49.875556 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.875528    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-dshm\") pod \"auth-test-model-98e305-kserve-fc69f58f9-g5mg2\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") " pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:26:49.875682 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.875587    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-tmp-dir\") pod \"auth-test-model-98e305-kserve-fc69f58f9-g5mg2\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") " pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:26:49.875682 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.875615    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-tls-certs\") pod \"auth-test-model-98e305-kserve-fc69f58f9-g5mg2\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") " pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:26:49.875860 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.875683    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-model-cache\") pod \"auth-test-model-98e305-kserve-fc69f58f9-g5mg2\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") " pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:26:49.875860 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.875768    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lqwk\" (UniqueName: \"kubernetes.io/projected/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-kube-api-access-9lqwk\") pod \"auth-test-model-98e305-kserve-fc69f58f9-g5mg2\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") " pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:26:49.875860 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.875813    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-home\") pod \"auth-test-model-98e305-kserve-fc69f58f9-g5mg2\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") " pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:26:49.977000 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.976972    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-dshm\") pod \"auth-test-model-98e305-kserve-fc69f58f9-g5mg2\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") " pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:26:49.977119 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.977015    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-tmp-dir\") pod \"auth-test-model-98e305-kserve-fc69f58f9-g5mg2\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") " pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:26:49.977119 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.977037    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-tls-certs\") pod \"auth-test-model-98e305-kserve-fc69f58f9-g5mg2\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") " pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:26:49.977119 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.977065    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-model-cache\") pod \"auth-test-model-98e305-kserve-fc69f58f9-g5mg2\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") " pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:26:49.977290 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.977150    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lqwk\" (UniqueName: \"kubernetes.io/projected/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-kube-api-access-9lqwk\") pod \"auth-test-model-98e305-kserve-fc69f58f9-g5mg2\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") " pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:26:49.977290 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.977203    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-home\") pod \"auth-test-model-98e305-kserve-fc69f58f9-g5mg2\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") " pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:26:49.977463 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.977443    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-tmp-dir\") pod \"auth-test-model-98e305-kserve-fc69f58f9-g5mg2\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") " pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:26:49.977537 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.977514    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-model-cache\") pod \"auth-test-model-98e305-kserve-fc69f58f9-g5mg2\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") " pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:26:49.977615 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.977595    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-home\") pod \"auth-test-model-98e305-kserve-fc69f58f9-g5mg2\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") " pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:26:49.979368 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.979346    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-dshm\") pod \"auth-test-model-98e305-kserve-fc69f58f9-g5mg2\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") " pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:26:49.979610 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.979595    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-tls-certs\") pod \"auth-test-model-98e305-kserve-fc69f58f9-g5mg2\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") " pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:26:49.986362 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:49.986329    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lqwk\" (UniqueName: \"kubernetes.io/projected/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-kube-api-access-9lqwk\") pod \"auth-test-model-98e305-kserve-fc69f58f9-g5mg2\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") " pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:26:50.116827 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:50.116804    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:26:50.124759 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:50.124743    2559 scope.go:117] "RemoveContainer" containerID="7f5b2c8c2b62b69a77bd92967001e2de9111770f936ada9782a1c3415f12fbaa"
Jul 17 19:26:50.261986 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:50.261962    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"]
Jul 17 19:26:50.264124 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:26:50.264099    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2f5f38a_2de7_4773_b4d5_66fcb47662e1.slice/crio-b98c22f62f06a5a6e78ea6bc6c773f7d8a842fe59a31c856a7781108e70661b1 WatchSource:0}: Error finding container b98c22f62f06a5a6e78ea6bc6c773f7d8a842fe59a31c856a7781108e70661b1: Status 404 returned error can't find the container with id b98c22f62f06a5a6e78ea6bc6c773f7d8a842fe59a31c856a7781108e70661b1
Jul 17 19:26:50.937364 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:50.937334    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2" event={"ID":"d2f5f38a-2de7-4773-b4d5-66fcb47662e1","Type":"ContainerStarted","Data":"df59e2e8bb289ca61270617ce2ad9379e912c4eacaa13dd06726ebcd0658bec0"}
Jul 17 19:26:50.937727 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:50.937384    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2" event={"ID":"d2f5f38a-2de7-4773-b4d5-66fcb47662e1","Type":"ContainerStarted","Data":"b98c22f62f06a5a6e78ea6bc6c773f7d8a842fe59a31c856a7781108e70661b1"}
Jul 17 19:26:50.939158 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:50.939137    2559 generic.go:356] "Generic (PLEG): container finished" podID="441bc1e2-a123-4824-a47b-8d52b7fed562" containerID="3504845ec35e9d7a0085604ff1370e7b4be0fb4a53590f4a3814c14f90ab7a51" exitCode=6
Jul 17 19:26:50.939243 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:50.939207    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc" event={"ID":"441bc1e2-a123-4824-a47b-8d52b7fed562","Type":"ContainerDied","Data":"3504845ec35e9d7a0085604ff1370e7b4be0fb4a53590f4a3814c14f90ab7a51"}
Jul 17 19:26:50.939281 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:50.939259    2559 scope.go:117] "RemoveContainer" containerID="7f5b2c8c2b62b69a77bd92967001e2de9111770f936ada9782a1c3415f12fbaa"
Jul 17 19:26:50.939571 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:50.939558    2559 scope.go:117] "RemoveContainer" containerID="3504845ec35e9d7a0085604ff1370e7b4be0fb4a53590f4a3814c14f90ab7a51"
Jul 17 19:26:50.939736 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:26:50.939718    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc_odh-ai-gateway-infra(441bc1e2-a123-4824-a47b-8d52b7fed562)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc" podUID="441bc1e2-a123-4824-a47b-8d52b7fed562"
Jul 17 19:26:50.955510 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:50.955468    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2" podStartSLOduration=1.955455888 podStartE2EDuration="1.955455888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:26:50.954606606 +0000 UTC m=+3221.396115287" watchObservedRunningTime="2026-07-17 19:26:50.955455888 +0000 UTC m=+3221.396964577"
Jul 17 19:26:51.981162 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:51.981128    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc"]
Jul 17 19:26:52.115213 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:52.115187    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc"
Jul 17 19:26:52.195593 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:52.195563    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/441bc1e2-a123-4824-a47b-8d52b7fed562-maas-api-service-ca\") pod \"441bc1e2-a123-4824-a47b-8d52b7fed562\" (UID: \"441bc1e2-a123-4824-a47b-8d52b7fed562\") "
Jul 17 19:26:52.195897 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:52.195877    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441bc1e2-a123-4824-a47b-8d52b7fed562-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "441bc1e2-a123-4824-a47b-8d52b7fed562" (UID: "441bc1e2-a123-4824-a47b-8d52b7fed562"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:26:52.296629 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:52.296572    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/441bc1e2-a123-4824-a47b-8d52b7fed562-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:26:52.954035 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:52.954011    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc"
Jul 17 19:26:52.954189 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:52.954060    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc" event={"ID":"441bc1e2-a123-4824-a47b-8d52b7fed562","Type":"ContainerDied","Data":"efb3e7bc3dec952e74acc609ca536a65e05f6abc8dcd99ccad1ee53355b0848a"}
Jul 17 19:26:52.954189 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:52.954141    2559 scope.go:117] "RemoveContainer" containerID="3504845ec35e9d7a0085604ff1370e7b4be0fb4a53590f4a3814c14f90ab7a51"
Jul 17 19:26:52.976904 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:52.976881    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc"]
Jul 17 19:26:52.981874 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:52.981851    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-hzhlc"]
Jul 17 19:26:54.125203 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:54.125170    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="441bc1e2-a123-4824-a47b-8d52b7fed562" path="/var/lib/kubelet/pods/441bc1e2-a123-4824-a47b-8d52b7fed562/volumes"
Jul 17 19:26:58.119854 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:58.119827    2559 scope.go:117] "RemoveContainer" containerID="f8ce99635915d2d1bf76098b938c089de33de97e08e3cb6e10d9fc7ba3a08356"
Jul 17 19:26:58.992083 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:58.992006    2559 generic.go:356] "Generic (PLEG): container finished" podID="014e23b1-4e20-4bba-9048-d998ed6e52b3" containerID="a91c8502ddfe5114fb09a44e257372e72ce0b47ba2c24a87afc11f1b3db3e452" exitCode=6
Jul 17 19:26:58.992369 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:58.992082    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf" event={"ID":"014e23b1-4e20-4bba-9048-d998ed6e52b3","Type":"ContainerDied","Data":"a91c8502ddfe5114fb09a44e257372e72ce0b47ba2c24a87afc11f1b3db3e452"}
Jul 17 19:26:58.992369 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:58.992140    2559 scope.go:117] "RemoveContainer" containerID="f8ce99635915d2d1bf76098b938c089de33de97e08e3cb6e10d9fc7ba3a08356"
Jul 17 19:26:58.992488 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:26:58.992415    2559 scope.go:117] "RemoveContainer" containerID="a91c8502ddfe5114fb09a44e257372e72ce0b47ba2c24a87afc11f1b3db3e452"
Jul 17 19:26:58.992651 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:26:58.992630    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf_odh-ai-gateway-infra(014e23b1-4e20-4bba-9048-d998ed6e52b3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf" podUID="014e23b1-4e20-4bba-9048-d998ed6e52b3"
Jul 17 19:27:00.018902 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:00.018870    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf"]
Jul 17 19:27:00.117503 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:00.117475    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:27:00.117646 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:00.117523    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:27:00.139830 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:00.139801    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:27:00.158373 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:00.158355    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf"
Jul 17 19:27:00.262026 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:00.262001    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/014e23b1-4e20-4bba-9048-d998ed6e52b3-maas-api-service-ca\") pod \"014e23b1-4e20-4bba-9048-d998ed6e52b3\" (UID: \"014e23b1-4e20-4bba-9048-d998ed6e52b3\") "
Jul 17 19:27:00.262364 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:00.262343    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014e23b1-4e20-4bba-9048-d998ed6e52b3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "014e23b1-4e20-4bba-9048-d998ed6e52b3" (UID: "014e23b1-4e20-4bba-9048-d998ed6e52b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:27:00.363163 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:00.363139    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/014e23b1-4e20-4bba-9048-d998ed6e52b3-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:27:01.006270 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.006238    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf"
Jul 17 19:27:01.006498 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.006278    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf" event={"ID":"014e23b1-4e20-4bba-9048-d998ed6e52b3","Type":"ContainerDied","Data":"6a4cea8f8e2e30bf8b7888a8edf210f9705157f5040ae4af90a26ceb311fd44b"}
Jul 17 19:27:01.006498 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.006378    2559 scope.go:117] "RemoveContainer" containerID="a91c8502ddfe5114fb09a44e257372e72ce0b47ba2c24a87afc11f1b3db3e452"
Jul 17 19:27:01.018144 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.018121    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:27:01.030486 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.030463    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf"]
Jul 17 19:27:01.034259 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.034227    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-6hgwf"]
Jul 17 19:27:01.440838 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.440808    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn"]
Jul 17 19:27:01.441193 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.441180    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="441bc1e2-a123-4824-a47b-8d52b7fed562" containerName="revoke-keys"
Jul 17 19:27:01.441240 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.441194    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="441bc1e2-a123-4824-a47b-8d52b7fed562" containerName="revoke-keys"
Jul 17 19:27:01.441240 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.441203    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="014e23b1-4e20-4bba-9048-d998ed6e52b3" containerName="revoke-keys"
Jul 17 19:27:01.441240 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.441209    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="014e23b1-4e20-4bba-9048-d998ed6e52b3" containerName="revoke-keys"
Jul 17 19:27:01.441240 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.441220    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="014e23b1-4e20-4bba-9048-d998ed6e52b3" containerName="revoke-keys"
Jul 17 19:27:01.441240 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.441226    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="014e23b1-4e20-4bba-9048-d998ed6e52b3" containerName="revoke-keys"
Jul 17 19:27:01.441240 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.441235    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="014e23b1-4e20-4bba-9048-d998ed6e52b3" containerName="revoke-keys"
Jul 17 19:27:01.441240 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.441240    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="014e23b1-4e20-4bba-9048-d998ed6e52b3" containerName="revoke-keys"
Jul 17 19:27:01.441494 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.441251    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="441bc1e2-a123-4824-a47b-8d52b7fed562" containerName="revoke-keys"
Jul 17 19:27:01.441494 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.441257    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="441bc1e2-a123-4824-a47b-8d52b7fed562" containerName="revoke-keys"
Jul 17 19:27:01.441494 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.441356    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="441bc1e2-a123-4824-a47b-8d52b7fed562" containerName="revoke-keys"
Jul 17 19:27:01.441494 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.441367    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="441bc1e2-a123-4824-a47b-8d52b7fed562" containerName="revoke-keys"
Jul 17 19:27:01.441494 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.441380    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="014e23b1-4e20-4bba-9048-d998ed6e52b3" containerName="revoke-keys"
Jul 17 19:27:01.441494 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.441389    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="014e23b1-4e20-4bba-9048-d998ed6e52b3" containerName="revoke-keys"
Jul 17 19:27:01.441494 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.441397    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="014e23b1-4e20-4bba-9048-d998ed6e52b3" containerName="revoke-keys"
Jul 17 19:27:01.445872 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.445857    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn"
Jul 17 19:27:01.448511 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.448490    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:27:01.451480 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.451458    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn"]
Jul 17 19:27:01.571757 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.571728    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c24074-c9f4-4c37-8461-330a41b2f30a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn\" (UID: \"c8c24074-c9f4-4c37-8461-330a41b2f30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn"
Jul 17 19:27:01.672580 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.672553    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c24074-c9f4-4c37-8461-330a41b2f30a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn\" (UID: \"c8c24074-c9f4-4c37-8461-330a41b2f30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn"
Jul 17 19:27:01.673115 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.673095    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c24074-c9f4-4c37-8461-330a41b2f30a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn\" (UID: \"c8c24074-c9f4-4c37-8461-330a41b2f30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn"
Jul 17 19:27:01.758613 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.758553    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn"
Jul 17 19:27:01.892027 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:01.891999    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn"]
Jul 17 19:27:01.893113 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:27:01.893089    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c24074_c9f4_4c37_8461_330a41b2f30a.slice/crio-52116fe4939f745d733f3ada6bb18049cb9ff6b731bba458b74688670bebe446 WatchSource:0}: Error finding container 52116fe4939f745d733f3ada6bb18049cb9ff6b731bba458b74688670bebe446: Status 404 returned error can't find the container with id 52116fe4939f745d733f3ada6bb18049cb9ff6b731bba458b74688670bebe446
Jul 17 19:27:02.013179 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:02.013096    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn" event={"ID":"c8c24074-c9f4-4c37-8461-330a41b2f30a","Type":"ContainerStarted","Data":"52116fe4939f745d733f3ada6bb18049cb9ff6b731bba458b74688670bebe446"}
Jul 17 19:27:02.126168 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:02.126137    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="014e23b1-4e20-4bba-9048-d998ed6e52b3" path="/var/lib/kubelet/pods/014e23b1-4e20-4bba-9048-d998ed6e52b3/volumes"
Jul 17 19:27:03.021277 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:03.021242    2559 generic.go:356] "Generic (PLEG): container finished" podID="c8c24074-c9f4-4c37-8461-330a41b2f30a" containerID="52b040351083b3469eb52cd1d69950e54740cc9cec621c1cc29f9dc1085e33fc" exitCode=6
Jul 17 19:27:03.021506 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:03.021339    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn" event={"ID":"c8c24074-c9f4-4c37-8461-330a41b2f30a","Type":"ContainerDied","Data":"52b040351083b3469eb52cd1d69950e54740cc9cec621c1cc29f9dc1085e33fc"}
Jul 17 19:27:03.021506 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:03.021500    2559 scope.go:117] "RemoveContainer" containerID="52b040351083b3469eb52cd1d69950e54740cc9cec621c1cc29f9dc1085e33fc"
Jul 17 19:27:04.032249 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:04.032157    2559 generic.go:356] "Generic (PLEG): container finished" podID="c8c24074-c9f4-4c37-8461-330a41b2f30a" containerID="2d1bf83af4f9407edd2ed0b01b30b753f8303d034897361dad5d03c316cf691c" exitCode=6
Jul 17 19:27:04.032688 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:04.032242    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn" event={"ID":"c8c24074-c9f4-4c37-8461-330a41b2f30a","Type":"ContainerDied","Data":"2d1bf83af4f9407edd2ed0b01b30b753f8303d034897361dad5d03c316cf691c"}
Jul 17 19:27:04.032688 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:04.032329    2559 scope.go:117] "RemoveContainer" containerID="52b040351083b3469eb52cd1d69950e54740cc9cec621c1cc29f9dc1085e33fc"
Jul 17 19:27:04.032688 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:04.032411    2559 scope.go:117] "RemoveContainer" containerID="2d1bf83af4f9407edd2ed0b01b30b753f8303d034897361dad5d03c316cf691c"
Jul 17 19:27:04.032688 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:27:04.032605    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn_odh-ai-gateway-infra(c8c24074-c9f4-4c37-8461-330a41b2f30a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn" podUID="c8c24074-c9f4-4c37-8461-330a41b2f30a"
Jul 17 19:27:05.039680 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:05.039654    2559 scope.go:117] "RemoveContainer" containerID="2d1bf83af4f9407edd2ed0b01b30b753f8303d034897361dad5d03c316cf691c"
Jul 17 19:27:05.040087 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:27:05.039837    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn_odh-ai-gateway-infra(c8c24074-c9f4-4c37-8461-330a41b2f30a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn" podUID="c8c24074-c9f4-4c37-8461-330a41b2f30a"
Jul 17 19:27:12.607907 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.607870    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"]
Jul 17 19:27:12.608492 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.608470    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="441bc1e2-a123-4824-a47b-8d52b7fed562" containerName="revoke-keys"
Jul 17 19:27:12.608596 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.608494    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="441bc1e2-a123-4824-a47b-8d52b7fed562" containerName="revoke-keys"
Jul 17 19:27:12.608654 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.608626    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="441bc1e2-a123-4824-a47b-8d52b7fed562" containerName="revoke-keys"
Jul 17 19:27:12.613374 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.613353    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:12.617206 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.617181    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2109fd\"/\"openshift-service-ca.crt\""
Jul 17 19:27:12.618189 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.618167    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2109fd\"/\"auth-test-model-2109fd-kserve-self-signed-certs\""
Jul 17 19:27:12.618446 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.618429    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2109fd\"/\"default-dockercfg-zf2rg\""
Jul 17 19:27:12.618542 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.618517    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2109fd\"/\"kube-root-ca.crt\""
Jul 17 19:27:12.620281 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.620203    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"]
Jul 17 19:27:12.767766 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.767738    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrxl2\" (UniqueName: \"kubernetes.io/projected/47c1f092-722e-43d5-93aa-f6d5a64572f3-kube-api-access-jrxl2\") pod \"auth-test-model-2109fd-kserve-6ccb948499-2bx86\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") " pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:12.767879 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.767785    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-home\") pod \"auth-test-model-2109fd-kserve-6ccb948499-2bx86\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") " pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:12.767879 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.767853    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-model-cache\") pod \"auth-test-model-2109fd-kserve-6ccb948499-2bx86\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") " pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:12.767879 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.767875    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47c1f092-722e-43d5-93aa-f6d5a64572f3-tls-certs\") pod \"auth-test-model-2109fd-kserve-6ccb948499-2bx86\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") " pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:12.768035 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.767917    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-tmp-dir\") pod \"auth-test-model-2109fd-kserve-6ccb948499-2bx86\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") " pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:12.768035 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.767946    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-dshm\") pod \"auth-test-model-2109fd-kserve-6ccb948499-2bx86\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") " pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:12.868650 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.868590    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-home\") pod \"auth-test-model-2109fd-kserve-6ccb948499-2bx86\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") " pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:12.868650 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.868624    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-model-cache\") pod \"auth-test-model-2109fd-kserve-6ccb948499-2bx86\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") " pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:12.868650 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.868641    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47c1f092-722e-43d5-93aa-f6d5a64572f3-tls-certs\") pod \"auth-test-model-2109fd-kserve-6ccb948499-2bx86\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") " pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:12.868876 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.868666    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-tmp-dir\") pod \"auth-test-model-2109fd-kserve-6ccb948499-2bx86\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") " pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:12.868876 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.868691    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-dshm\") pod \"auth-test-model-2109fd-kserve-6ccb948499-2bx86\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") " pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:12.868876 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.868750    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrxl2\" (UniqueName: \"kubernetes.io/projected/47c1f092-722e-43d5-93aa-f6d5a64572f3-kube-api-access-jrxl2\") pod \"auth-test-model-2109fd-kserve-6ccb948499-2bx86\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") " pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:12.869049 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.869014    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-home\") pod \"auth-test-model-2109fd-kserve-6ccb948499-2bx86\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") " pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:12.869111 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.869037    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-model-cache\") pod \"auth-test-model-2109fd-kserve-6ccb948499-2bx86\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") " pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:12.869111 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.869092    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-tmp-dir\") pod \"auth-test-model-2109fd-kserve-6ccb948499-2bx86\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") " pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:12.871321 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.871279    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-dshm\") pod \"auth-test-model-2109fd-kserve-6ccb948499-2bx86\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") " pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:12.871595 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.871572    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47c1f092-722e-43d5-93aa-f6d5a64572f3-tls-certs\") pod \"auth-test-model-2109fd-kserve-6ccb948499-2bx86\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") " pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:12.877237 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.877217    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrxl2\" (UniqueName: \"kubernetes.io/projected/47c1f092-722e-43d5-93aa-f6d5a64572f3-kube-api-access-jrxl2\") pod \"auth-test-model-2109fd-kserve-6ccb948499-2bx86\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") " pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:12.925435 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:12.925411    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:13.062601 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:13.062572    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"]
Jul 17 19:27:13.063916 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:27:13.063885    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47c1f092_722e_43d5_93aa_f6d5a64572f3.slice/crio-6d50253a37d88de94dbaa9df2f70e60e48a145bb45459177b5d074ca2371fb74 WatchSource:0}: Error finding container 6d50253a37d88de94dbaa9df2f70e60e48a145bb45459177b5d074ca2371fb74: Status 404 returned error can't find the container with id 6d50253a37d88de94dbaa9df2f70e60e48a145bb45459177b5d074ca2371fb74
Jul 17 19:27:13.089360 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:13.089329    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86" event={"ID":"47c1f092-722e-43d5-93aa-f6d5a64572f3","Type":"ContainerStarted","Data":"6d50253a37d88de94dbaa9df2f70e60e48a145bb45459177b5d074ca2371fb74"}
Jul 17 19:27:14.096956 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:14.096924    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86" event={"ID":"47c1f092-722e-43d5-93aa-f6d5a64572f3","Type":"ContainerStarted","Data":"0ac255dbea50b12c53bf5bfdeb63260fbecccfca23ffab17ffe2a9787b1d3577"}
Jul 17 19:27:14.115793 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:14.115751    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86" podStartSLOduration=2.115735274 podStartE2EDuration="2.115735274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:27:14.112570751 +0000 UTC m=+3244.554079442" watchObservedRunningTime="2026-07-17 19:27:14.115735274 +0000 UTC m=+3244.557243967"
Jul 17 19:27:19.120267 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:19.120233    2559 scope.go:117] "RemoveContainer" containerID="2d1bf83af4f9407edd2ed0b01b30b753f8303d034897361dad5d03c316cf691c"
Jul 17 19:27:20.138153 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:20.138121    2559 generic.go:356] "Generic (PLEG): container finished" podID="c8c24074-c9f4-4c37-8461-330a41b2f30a" containerID="6beb3ee5d5c01051e20065722895e0a92cc6a7fddfaee3d5d63f83fd5acd01b8" exitCode=6
Jul 17 19:27:20.138652 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:20.138174    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn" event={"ID":"c8c24074-c9f4-4c37-8461-330a41b2f30a","Type":"ContainerDied","Data":"6beb3ee5d5c01051e20065722895e0a92cc6a7fddfaee3d5d63f83fd5acd01b8"}
Jul 17 19:27:20.138652 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:20.138220    2559 scope.go:117] "RemoveContainer" containerID="2d1bf83af4f9407edd2ed0b01b30b753f8303d034897361dad5d03c316cf691c"
Jul 17 19:27:20.138652 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:20.138569    2559 scope.go:117] "RemoveContainer" containerID="6beb3ee5d5c01051e20065722895e0a92cc6a7fddfaee3d5d63f83fd5acd01b8"
Jul 17 19:27:20.138840 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:27:20.138818    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn_odh-ai-gateway-infra(c8c24074-c9f4-4c37-8461-330a41b2f30a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn" podUID="c8c24074-c9f4-4c37-8461-330a41b2f30a"
Jul 17 19:27:21.165856 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:21.165827    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn"]
Jul 17 19:27:21.294620 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:21.294588    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn"
Jul 17 19:27:21.345955 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:21.345930    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c24074-c9f4-4c37-8461-330a41b2f30a-maas-api-service-ca\") pod \"c8c24074-c9f4-4c37-8461-330a41b2f30a\" (UID: \"c8c24074-c9f4-4c37-8461-330a41b2f30a\") "
Jul 17 19:27:21.346267 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:21.346246    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c24074-c9f4-4c37-8461-330a41b2f30a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c8c24074-c9f4-4c37-8461-330a41b2f30a" (UID: "c8c24074-c9f4-4c37-8461-330a41b2f30a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:27:21.447238 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:21.447176    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c24074-c9f4-4c37-8461-330a41b2f30a-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:27:22.152596 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:22.152564    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn"
Jul 17 19:27:22.152596 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:22.152578    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn" event={"ID":"c8c24074-c9f4-4c37-8461-330a41b2f30a","Type":"ContainerDied","Data":"52116fe4939f745d733f3ada6bb18049cb9ff6b731bba458b74688670bebe446"}
Jul 17 19:27:22.152802 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:22.152639    2559 scope.go:117] "RemoveContainer" containerID="6beb3ee5d5c01051e20065722895e0a92cc6a7fddfaee3d5d63f83fd5acd01b8"
Jul 17 19:27:22.172253 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:22.172230    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn"]
Jul 17 19:27:22.184032 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:22.184008    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-kdxjn"]
Jul 17 19:27:22.926067 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:22.926035    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:22.926246 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:22.926086    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:22.938359 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:22.938327    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:23.171013 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:23.170988    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:27:24.126165 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:27:24.126135    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8c24074-c9f4-4c37-8461-330a41b2f30a" path="/var/lib/kubelet/pods/c8c24074-c9f4-4c37-8461-330a41b2f30a/volumes"
Jul 17 19:28:01.483663 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:01.483630    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67"]
Jul 17 19:28:01.484104 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:01.484009    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8c24074-c9f4-4c37-8461-330a41b2f30a" containerName="revoke-keys"
Jul 17 19:28:01.484104 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:01.484021    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8c24074-c9f4-4c37-8461-330a41b2f30a" containerName="revoke-keys"
Jul 17 19:28:01.484104 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:01.484045    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8c24074-c9f4-4c37-8461-330a41b2f30a" containerName="revoke-keys"
Jul 17 19:28:01.484104 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:01.484051    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8c24074-c9f4-4c37-8461-330a41b2f30a" containerName="revoke-keys"
Jul 17 19:28:01.484238 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:01.484107    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8c24074-c9f4-4c37-8461-330a41b2f30a" containerName="revoke-keys"
Jul 17 19:28:01.484238 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:01.484115    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8c24074-c9f4-4c37-8461-330a41b2f30a" containerName="revoke-keys"
Jul 17 19:28:01.484238 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:01.484123    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8c24074-c9f4-4c37-8461-330a41b2f30a" containerName="revoke-keys"
Jul 17 19:28:01.487294 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:01.487277    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67"
Jul 17 19:28:01.489678 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:01.489656    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:28:01.494435 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:01.494403    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67"]
Jul 17 19:28:01.598499 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:01.598467    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f78e9b-0d57-4ae3-8e17-bd3e08216bda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-dpw67\" (UID: \"e5f78e9b-0d57-4ae3-8e17-bd3e08216bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67"
Jul 17 19:28:01.699539 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:01.699502    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f78e9b-0d57-4ae3-8e17-bd3e08216bda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-dpw67\" (UID: \"e5f78e9b-0d57-4ae3-8e17-bd3e08216bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67"
Jul 17 19:28:01.700109 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:01.700089    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f78e9b-0d57-4ae3-8e17-bd3e08216bda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-dpw67\" (UID: \"e5f78e9b-0d57-4ae3-8e17-bd3e08216bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67"
Jul 17 19:28:01.799265 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:01.799197    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67"
Jul 17 19:28:01.930150 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:01.930126    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67"]
Jul 17 19:28:01.931626 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:28:01.931600    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f78e9b_0d57_4ae3_8e17_bd3e08216bda.slice/crio-a70a2d6eb8ad246f35c8a0a2f1349ed09ac64802a0cf79beee8cc04c1b294859 WatchSource:0}: Error finding container a70a2d6eb8ad246f35c8a0a2f1349ed09ac64802a0cf79beee8cc04c1b294859: Status 404 returned error can't find the container with id a70a2d6eb8ad246f35c8a0a2f1349ed09ac64802a0cf79beee8cc04c1b294859
Jul 17 19:28:02.389491 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:02.389454    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67" event={"ID":"e5f78e9b-0d57-4ae3-8e17-bd3e08216bda","Type":"ContainerStarted","Data":"587db5488c5decfd515c5d8b82d48ab4247160bf386e791b5ef3bd6108f0d6f1"}
Jul 17 19:28:02.389655 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:02.389511    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67" event={"ID":"e5f78e9b-0d57-4ae3-8e17-bd3e08216bda","Type":"ContainerStarted","Data":"a70a2d6eb8ad246f35c8a0a2f1349ed09ac64802a0cf79beee8cc04c1b294859"}
Jul 17 19:28:02.407213 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:02.407129    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67" podStartSLOduration=1.4071153220000001 podStartE2EDuration="1.407115322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:28:02.404911849 +0000 UTC m=+3292.846420539" watchObservedRunningTime="2026-07-17 19:28:02.407115322 +0000 UTC m=+3292.848624010"
Jul 17 19:28:03.397289 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:03.397252    2559 generic.go:356] "Generic (PLEG): container finished" podID="e5f78e9b-0d57-4ae3-8e17-bd3e08216bda" containerID="587db5488c5decfd515c5d8b82d48ab4247160bf386e791b5ef3bd6108f0d6f1" exitCode=6
Jul 17 19:28:03.397687 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:03.397327    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67" event={"ID":"e5f78e9b-0d57-4ae3-8e17-bd3e08216bda","Type":"ContainerDied","Data":"587db5488c5decfd515c5d8b82d48ab4247160bf386e791b5ef3bd6108f0d6f1"}
Jul 17 19:28:03.397687 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:03.397556    2559 scope.go:117] "RemoveContainer" containerID="587db5488c5decfd515c5d8b82d48ab4247160bf386e791b5ef3bd6108f0d6f1"
Jul 17 19:28:04.404864 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:04.404831    2559 generic.go:356] "Generic (PLEG): container finished" podID="e5f78e9b-0d57-4ae3-8e17-bd3e08216bda" containerID="d17793f5e39823a3741cb3f6bea8a81110b5242ae0e9c2e205722f32b67bfc98" exitCode=6
Jul 17 19:28:04.405278 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:04.404915    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67" event={"ID":"e5f78e9b-0d57-4ae3-8e17-bd3e08216bda","Type":"ContainerDied","Data":"d17793f5e39823a3741cb3f6bea8a81110b5242ae0e9c2e205722f32b67bfc98"}
Jul 17 19:28:04.405278 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:04.404973    2559 scope.go:117] "RemoveContainer" containerID="587db5488c5decfd515c5d8b82d48ab4247160bf386e791b5ef3bd6108f0d6f1"
Jul 17 19:28:04.405278 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:04.405119    2559 scope.go:117] "RemoveContainer" containerID="d17793f5e39823a3741cb3f6bea8a81110b5242ae0e9c2e205722f32b67bfc98"
Jul 17 19:28:04.405433 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:28:04.405380    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-dpw67_odh-ai-gateway-infra(e5f78e9b-0d57-4ae3-8e17-bd3e08216bda)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67" podUID="e5f78e9b-0d57-4ae3-8e17-bd3e08216bda"
Jul 17 19:28:05.413171 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:05.413141    2559 scope.go:117] "RemoveContainer" containerID="d17793f5e39823a3741cb3f6bea8a81110b5242ae0e9c2e205722f32b67bfc98"
Jul 17 19:28:05.413610 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:28:05.413375    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-dpw67_odh-ai-gateway-infra(e5f78e9b-0d57-4ae3-8e17-bd3e08216bda)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67" podUID="e5f78e9b-0d57-4ae3-8e17-bd3e08216bda"
Jul 17 19:28:06.165002 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:06.164971    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd"]
Jul 17 19:28:06.165381 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:06.165368    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8c24074-c9f4-4c37-8461-330a41b2f30a" containerName="revoke-keys"
Jul 17 19:28:06.165445 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:06.165383    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8c24074-c9f4-4c37-8461-330a41b2f30a" containerName="revoke-keys"
Jul 17 19:28:06.169614 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:06.169598    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd"
Jul 17 19:28:06.176411 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:06.176372    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd"]
Jul 17 19:28:06.341939 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:06.341895    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/882ca181-6d52-45a3-a989-fac488a2813a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd\" (UID: \"882ca181-6d52-45a3-a989-fac488a2813a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd"
Jul 17 19:28:06.443052 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:06.442981    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/882ca181-6d52-45a3-a989-fac488a2813a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd\" (UID: \"882ca181-6d52-45a3-a989-fac488a2813a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd"
Jul 17 19:28:06.443588 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:06.443569    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/882ca181-6d52-45a3-a989-fac488a2813a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd\" (UID: \"882ca181-6d52-45a3-a989-fac488a2813a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd"
Jul 17 19:28:06.482037 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:06.482004    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd"
Jul 17 19:28:06.638027 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:06.638005    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd"]
Jul 17 19:28:06.640486 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:28:06.640452    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882ca181_6d52_45a3_a989_fac488a2813a.slice/crio-c88abef0212815f0d607cd6aaa06d88be37da887487c6c71ead35fc6ac297524 WatchSource:0}: Error finding container c88abef0212815f0d607cd6aaa06d88be37da887487c6c71ead35fc6ac297524: Status 404 returned error can't find the container with id c88abef0212815f0d607cd6aaa06d88be37da887487c6c71ead35fc6ac297524
Jul 17 19:28:07.426589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:07.426555    2559 generic.go:356] "Generic (PLEG): container finished" podID="882ca181-6d52-45a3-a989-fac488a2813a" containerID="65db12e9dabb9875a51055b7dfc6c1da7b430f9afb8b13963efb0e85361fc50b" exitCode=6
Jul 17 19:28:07.426720 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:07.426625    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd" event={"ID":"882ca181-6d52-45a3-a989-fac488a2813a","Type":"ContainerDied","Data":"65db12e9dabb9875a51055b7dfc6c1da7b430f9afb8b13963efb0e85361fc50b"}
Jul 17 19:28:07.426720 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:07.426699    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd" event={"ID":"882ca181-6d52-45a3-a989-fac488a2813a","Type":"ContainerStarted","Data":"c88abef0212815f0d607cd6aaa06d88be37da887487c6c71ead35fc6ac297524"}
Jul 17 19:28:07.426894 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:07.426882    2559 scope.go:117] "RemoveContainer" containerID="65db12e9dabb9875a51055b7dfc6c1da7b430f9afb8b13963efb0e85361fc50b"
Jul 17 19:28:08.434219 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:08.434187    2559 generic.go:356] "Generic (PLEG): container finished" podID="882ca181-6d52-45a3-a989-fac488a2813a" containerID="b75555f2411460852696156d780a8ad652ad81aa6899d386a525a62fe7ab84ac" exitCode=6
Jul 17 19:28:08.434714 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:08.434272    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd" event={"ID":"882ca181-6d52-45a3-a989-fac488a2813a","Type":"ContainerDied","Data":"b75555f2411460852696156d780a8ad652ad81aa6899d386a525a62fe7ab84ac"}
Jul 17 19:28:08.434714 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:08.434357    2559 scope.go:117] "RemoveContainer" containerID="65db12e9dabb9875a51055b7dfc6c1da7b430f9afb8b13963efb0e85361fc50b"
Jul 17 19:28:08.434714 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:08.434538    2559 scope.go:117] "RemoveContainer" containerID="b75555f2411460852696156d780a8ad652ad81aa6899d386a525a62fe7ab84ac"
Jul 17 19:28:08.434869 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:28:08.434816    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd_odh-ai-gateway-infra(882ca181-6d52-45a3-a989-fac488a2813a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd" podUID="882ca181-6d52-45a3-a989-fac488a2813a"
Jul 17 19:28:09.441521 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:09.441494    2559 scope.go:117] "RemoveContainer" containerID="b75555f2411460852696156d780a8ad652ad81aa6899d386a525a62fe7ab84ac"
Jul 17 19:28:09.441989 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:28:09.441809    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd_odh-ai-gateway-infra(882ca181-6d52-45a3-a989-fac488a2813a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd" podUID="882ca181-6d52-45a3-a989-fac488a2813a"
Jul 17 19:28:10.439154 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:10.439125    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 19:28:10.444811 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:10.444784    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 19:28:10.447064 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:10.447043    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 19:28:10.452365 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:10.452350    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 19:28:11.536366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:11.536334    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-njxp6"]
Jul 17 19:28:11.541590 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:11.541571    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-njxp6"
Jul 17 19:28:11.544702 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:11.544668    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-r9nzb\""
Jul 17 19:28:11.547457 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:11.547436    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-njxp6"]
Jul 17 19:28:11.581973 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:11.581952    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8xwx\" (UniqueName: \"kubernetes.io/projected/a585ef93-4f77-404b-ae58-9183acd92a48-kube-api-access-s8xwx\") pod \"kuadrant-operator-catalog-njxp6\" (UID: \"a585ef93-4f77-404b-ae58-9183acd92a48\") " pod="kuadrant-system/kuadrant-operator-catalog-njxp6"
Jul 17 19:28:11.682499 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:11.682468    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xwx\" (UniqueName: \"kubernetes.io/projected/a585ef93-4f77-404b-ae58-9183acd92a48-kube-api-access-s8xwx\") pod \"kuadrant-operator-catalog-njxp6\" (UID: \"a585ef93-4f77-404b-ae58-9183acd92a48\") " pod="kuadrant-system/kuadrant-operator-catalog-njxp6"
Jul 17 19:28:11.690526 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:11.690498    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xwx\" (UniqueName: \"kubernetes.io/projected/a585ef93-4f77-404b-ae58-9183acd92a48-kube-api-access-s8xwx\") pod \"kuadrant-operator-catalog-njxp6\" (UID: \"a585ef93-4f77-404b-ae58-9183acd92a48\") " pod="kuadrant-system/kuadrant-operator-catalog-njxp6"
Jul 17 19:28:11.853128 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:11.853088    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-njxp6"
Jul 17 19:28:11.988626 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:11.988602    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-njxp6"]
Jul 17 19:28:11.989075 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:28:11.989039    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda585ef93_4f77_404b_ae58_9183acd92a48.slice/crio-ad37ec7e6efeb52bcf8719f89f60b39e174497c757d4e61d6bc9e15d43559253 WatchSource:0}: Error finding container ad37ec7e6efeb52bcf8719f89f60b39e174497c757d4e61d6bc9e15d43559253: Status 404 returned error can't find the container with id ad37ec7e6efeb52bcf8719f89f60b39e174497c757d4e61d6bc9e15d43559253
Jul 17 19:28:11.990595 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:11.990575    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 19:28:12.459500 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:12.459470    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-njxp6" event={"ID":"a585ef93-4f77-404b-ae58-9183acd92a48","Type":"ContainerStarted","Data":"ad37ec7e6efeb52bcf8719f89f60b39e174497c757d4e61d6bc9e15d43559253"}
Jul 17 19:28:14.475023 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:14.474934    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-njxp6" event={"ID":"a585ef93-4f77-404b-ae58-9183acd92a48","Type":"ContainerStarted","Data":"a69cd39ec69b8de95488275d5cfdcae66b21143b5de176215e102c2c88e91fa3"}
Jul 17 19:28:14.494733 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:14.494683    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-njxp6" podStartSLOduration=1.305015519 podStartE2EDuration="3.494668087s" totalImagesPullingTime="2.189652568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:28:14.492285427 +0000 UTC m=+3304.933794116" watchObservedRunningTime="2026-07-17 19:28:14.494668087 +0000 UTC m=+3304.936176775"
Jul 17 19:28:16.119678 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:16.119647    2559 scope.go:117] "RemoveContainer" containerID="d17793f5e39823a3741cb3f6bea8a81110b5242ae0e9c2e205722f32b67bfc98"
Jul 17 19:28:16.487651 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:16.487616    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67" event={"ID":"e5f78e9b-0d57-4ae3-8e17-bd3e08216bda","Type":"ContainerStarted","Data":"a7a53cb961f404bcc4a60411ea5d58e98613e2013e850feb6f4303800e6341f4"}
Jul 17 19:28:17.494709 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:17.494676    2559 generic.go:356] "Generic (PLEG): container finished" podID="e5f78e9b-0d57-4ae3-8e17-bd3e08216bda" containerID="a7a53cb961f404bcc4a60411ea5d58e98613e2013e850feb6f4303800e6341f4" exitCode=6
Jul 17 19:28:17.495151 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:17.494730    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67" event={"ID":"e5f78e9b-0d57-4ae3-8e17-bd3e08216bda","Type":"ContainerDied","Data":"a7a53cb961f404bcc4a60411ea5d58e98613e2013e850feb6f4303800e6341f4"}
Jul 17 19:28:17.495151 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:17.494804    2559 scope.go:117] "RemoveContainer" containerID="d17793f5e39823a3741cb3f6bea8a81110b5242ae0e9c2e205722f32b67bfc98"
Jul 17 19:28:17.495151 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:17.495061    2559 scope.go:117] "RemoveContainer" containerID="a7a53cb961f404bcc4a60411ea5d58e98613e2013e850feb6f4303800e6341f4"
Jul 17 19:28:17.495395 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:28:17.495296    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-dpw67_odh-ai-gateway-infra(e5f78e9b-0d57-4ae3-8e17-bd3e08216bda)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67" podUID="e5f78e9b-0d57-4ae3-8e17-bd3e08216bda"
Jul 17 19:28:17.519670 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:17.519647    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67"]
Jul 17 19:28:18.648713 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:18.648686    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67"
Jul 17 19:28:18.744466 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:18.744433    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f78e9b-0d57-4ae3-8e17-bd3e08216bda-maas-api-service-ca\") pod \"e5f78e9b-0d57-4ae3-8e17-bd3e08216bda\" (UID: \"e5f78e9b-0d57-4ae3-8e17-bd3e08216bda\") "
Jul 17 19:28:18.744844 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:18.744825    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f78e9b-0d57-4ae3-8e17-bd3e08216bda-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5f78e9b-0d57-4ae3-8e17-bd3e08216bda" (UID: "e5f78e9b-0d57-4ae3-8e17-bd3e08216bda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:28:18.845551 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:18.845522    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f78e9b-0d57-4ae3-8e17-bd3e08216bda-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:28:19.512097 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:19.512072    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67"
Jul 17 19:28:19.512295 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:19.512109    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67" event={"ID":"e5f78e9b-0d57-4ae3-8e17-bd3e08216bda","Type":"ContainerDied","Data":"a70a2d6eb8ad246f35c8a0a2f1349ed09ac64802a0cf79beee8cc04c1b294859"}
Jul 17 19:28:19.512295 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:19.512166    2559 scope.go:117] "RemoveContainer" containerID="a7a53cb961f404bcc4a60411ea5d58e98613e2013e850feb6f4303800e6341f4"
Jul 17 19:28:19.535510 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:19.535476    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67"]
Jul 17 19:28:19.541131 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:19.541108    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-dpw67"]
Jul 17 19:28:20.125349 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:20.125299    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5f78e9b-0d57-4ae3-8e17-bd3e08216bda" path="/var/lib/kubelet/pods/e5f78e9b-0d57-4ae3-8e17-bd3e08216bda/volumes"
Jul 17 19:28:21.119785 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:21.119754    2559 scope.go:117] "RemoveContainer" containerID="b75555f2411460852696156d780a8ad652ad81aa6899d386a525a62fe7ab84ac"
Jul 17 19:28:21.526762 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:21.526724    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd" event={"ID":"882ca181-6d52-45a3-a989-fac488a2813a","Type":"ContainerStarted","Data":"df7c851056ba42ab119942d2743156ebbbc2eb33aac8e703e1b47a4d002f76c8"}
Jul 17 19:28:21.543816 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:21.543774    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd" podStartSLOduration=15.543760659 podStartE2EDuration="15.543760659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:28:21.541383299 +0000 UTC m=+3311.982891989" watchObservedRunningTime="2026-07-17 19:28:21.543760659 +0000 UTC m=+3311.985269347"
Jul 17 19:28:21.853595 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:21.853571    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-njxp6"
Jul 17 19:28:21.853723 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:21.853661    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-njxp6"
Jul 17 19:28:21.876466 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:21.876444    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-njxp6"
Jul 17 19:28:22.534133 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:22.534099    2559 generic.go:356] "Generic (PLEG): container finished" podID="882ca181-6d52-45a3-a989-fac488a2813a" containerID="df7c851056ba42ab119942d2743156ebbbc2eb33aac8e703e1b47a4d002f76c8" exitCode=6
Jul 17 19:28:22.534583 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:22.534171    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd" event={"ID":"882ca181-6d52-45a3-a989-fac488a2813a","Type":"ContainerDied","Data":"df7c851056ba42ab119942d2743156ebbbc2eb33aac8e703e1b47a4d002f76c8"}
Jul 17 19:28:22.534583 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:22.534238    2559 scope.go:117] "RemoveContainer" containerID="b75555f2411460852696156d780a8ad652ad81aa6899d386a525a62fe7ab84ac"
Jul 17 19:28:22.534711 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:22.534660    2559 scope.go:117] "RemoveContainer" containerID="df7c851056ba42ab119942d2743156ebbbc2eb33aac8e703e1b47a4d002f76c8"
Jul 17 19:28:22.534917 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:28:22.534893    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd_odh-ai-gateway-infra(882ca181-6d52-45a3-a989-fac488a2813a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd" podUID="882ca181-6d52-45a3-a989-fac488a2813a"
Jul 17 19:28:22.558629 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:22.558597    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd"]
Jul 17 19:28:22.560443 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:22.560418    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-njxp6"
Jul 17 19:28:22.599861 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:22.599833    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-njxp6"]
Jul 17 19:28:23.678452 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:23.678430    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd"
Jul 17 19:28:23.788897 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:23.788868    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/882ca181-6d52-45a3-a989-fac488a2813a-maas-api-service-ca\") pod \"882ca181-6d52-45a3-a989-fac488a2813a\" (UID: \"882ca181-6d52-45a3-a989-fac488a2813a\") "
Jul 17 19:28:23.789218 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:23.789197    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/882ca181-6d52-45a3-a989-fac488a2813a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "882ca181-6d52-45a3-a989-fac488a2813a" (UID: "882ca181-6d52-45a3-a989-fac488a2813a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:28:23.890550 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:23.890507    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/882ca181-6d52-45a3-a989-fac488a2813a-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:28:24.550944 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:24.550916    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd"
Jul 17 19:28:24.551117 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:24.550962    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd" event={"ID":"882ca181-6d52-45a3-a989-fac488a2813a","Type":"ContainerDied","Data":"c88abef0212815f0d607cd6aaa06d88be37da887487c6c71ead35fc6ac297524"}
Jul 17 19:28:24.551117 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:24.551014    2559 scope.go:117] "RemoveContainer" containerID="df7c851056ba42ab119942d2743156ebbbc2eb33aac8e703e1b47a4d002f76c8"
Jul 17 19:28:24.551244 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:24.551151    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-njxp6" podUID="a585ef93-4f77-404b-ae58-9183acd92a48" containerName="registry-server" containerID="cri-o://a69cd39ec69b8de95488275d5cfdcae66b21143b5de176215e102c2c88e91fa3" gracePeriod=2
Jul 17 19:28:24.567376 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:24.567351    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd"]
Jul 17 19:28:24.574669 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:24.574646    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-7zjmd"]
Jul 17 19:28:24.793041 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:24.793012    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-njxp6"
Jul 17 19:28:24.899133 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:24.899103    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8xwx\" (UniqueName: \"kubernetes.io/projected/a585ef93-4f77-404b-ae58-9183acd92a48-kube-api-access-s8xwx\") pod \"a585ef93-4f77-404b-ae58-9183acd92a48\" (UID: \"a585ef93-4f77-404b-ae58-9183acd92a48\") "
Jul 17 19:28:24.901258 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:24.901236    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a585ef93-4f77-404b-ae58-9183acd92a48-kube-api-access-s8xwx" (OuterVolumeSpecName: "kube-api-access-s8xwx") pod "a585ef93-4f77-404b-ae58-9183acd92a48" (UID: "a585ef93-4f77-404b-ae58-9183acd92a48"). InnerVolumeSpecName "kube-api-access-s8xwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:28:25.000085 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:25.000054    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8xwx\" (UniqueName: \"kubernetes.io/projected/a585ef93-4f77-404b-ae58-9183acd92a48-kube-api-access-s8xwx\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:28:25.558552 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:25.558515    2559 generic.go:356] "Generic (PLEG): container finished" podID="a585ef93-4f77-404b-ae58-9183acd92a48" containerID="a69cd39ec69b8de95488275d5cfdcae66b21143b5de176215e102c2c88e91fa3" exitCode=0
Jul 17 19:28:25.558748 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:25.558563    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-njxp6"
Jul 17 19:28:25.558748 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:25.558606    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-njxp6" event={"ID":"a585ef93-4f77-404b-ae58-9183acd92a48","Type":"ContainerDied","Data":"a69cd39ec69b8de95488275d5cfdcae66b21143b5de176215e102c2c88e91fa3"}
Jul 17 19:28:25.558748 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:25.558659    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-njxp6" event={"ID":"a585ef93-4f77-404b-ae58-9183acd92a48","Type":"ContainerDied","Data":"ad37ec7e6efeb52bcf8719f89f60b39e174497c757d4e61d6bc9e15d43559253"}
Jul 17 19:28:25.558748 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:25.558684    2559 scope.go:117] "RemoveContainer" containerID="a69cd39ec69b8de95488275d5cfdcae66b21143b5de176215e102c2c88e91fa3"
Jul 17 19:28:25.576088 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:25.576075    2559 scope.go:117] "RemoveContainer" containerID="a69cd39ec69b8de95488275d5cfdcae66b21143b5de176215e102c2c88e91fa3"
Jul 17 19:28:25.576354 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:28:25.576335    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a69cd39ec69b8de95488275d5cfdcae66b21143b5de176215e102c2c88e91fa3\": container with ID starting with a69cd39ec69b8de95488275d5cfdcae66b21143b5de176215e102c2c88e91fa3 not found: ID does not exist" containerID="a69cd39ec69b8de95488275d5cfdcae66b21143b5de176215e102c2c88e91fa3"
Jul 17 19:28:25.576418 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:25.576363    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a69cd39ec69b8de95488275d5cfdcae66b21143b5de176215e102c2c88e91fa3"} err="failed to get container status \"a69cd39ec69b8de95488275d5cfdcae66b21143b5de176215e102c2c88e91fa3\": rpc error: code = NotFound desc = could not find container \"a69cd39ec69b8de95488275d5cfdcae66b21143b5de176215e102c2c88e91fa3\": container with ID starting with a69cd39ec69b8de95488275d5cfdcae66b21143b5de176215e102c2c88e91fa3 not found: ID does not exist"
Jul 17 19:28:25.586020 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:25.585993    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-njxp6"]
Jul 17 19:28:25.589222 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:25.589198    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-njxp6"]
Jul 17 19:28:26.126098 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:26.126064    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="882ca181-6d52-45a3-a989-fac488a2813a" path="/var/lib/kubelet/pods/882ca181-6d52-45a3-a989-fac488a2813a/volumes"
Jul 17 19:28:26.126631 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:26.126598    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a585ef93-4f77-404b-ae58-9183acd92a48" path="/var/lib/kubelet/pods/a585ef93-4f77-404b-ae58-9183acd92a48/volumes"
Jul 17 19:28:51.524394 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.524363    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p"]
Jul 17 19:28:51.524824 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.524733    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="882ca181-6d52-45a3-a989-fac488a2813a" containerName="revoke-keys"
Jul 17 19:28:51.524824 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.524744    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="882ca181-6d52-45a3-a989-fac488a2813a" containerName="revoke-keys"
Jul 17 19:28:51.524824 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.524755    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="882ca181-6d52-45a3-a989-fac488a2813a" containerName="revoke-keys"
Jul 17 19:28:51.524824 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.524760    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="882ca181-6d52-45a3-a989-fac488a2813a" containerName="revoke-keys"
Jul 17 19:28:51.524824 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.524769    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5f78e9b-0d57-4ae3-8e17-bd3e08216bda" containerName="revoke-keys"
Jul 17 19:28:51.524824 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.524774    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5f78e9b-0d57-4ae3-8e17-bd3e08216bda" containerName="revoke-keys"
Jul 17 19:28:51.524824 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.524789    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a585ef93-4f77-404b-ae58-9183acd92a48" containerName="registry-server"
Jul 17 19:28:51.524824 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.524794    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a585ef93-4f77-404b-ae58-9183acd92a48" containerName="registry-server"
Jul 17 19:28:51.524824 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.524802    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5f78e9b-0d57-4ae3-8e17-bd3e08216bda" containerName="revoke-keys"
Jul 17 19:28:51.524824 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.524807    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5f78e9b-0d57-4ae3-8e17-bd3e08216bda" containerName="revoke-keys"
Jul 17 19:28:51.525142 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.524869    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5f78e9b-0d57-4ae3-8e17-bd3e08216bda" containerName="revoke-keys"
Jul 17 19:28:51.525142 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.524876    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5f78e9b-0d57-4ae3-8e17-bd3e08216bda" containerName="revoke-keys"
Jul 17 19:28:51.525142 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.524880    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a585ef93-4f77-404b-ae58-9183acd92a48" containerName="registry-server"
Jul 17 19:28:51.525142 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.524887    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="882ca181-6d52-45a3-a989-fac488a2813a" containerName="revoke-keys"
Jul 17 19:28:51.525142 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.524893    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="882ca181-6d52-45a3-a989-fac488a2813a" containerName="revoke-keys"
Jul 17 19:28:51.525142 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.524899    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="882ca181-6d52-45a3-a989-fac488a2813a" containerName="revoke-keys"
Jul 17 19:28:51.528266 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.528247    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p"
Jul 17 19:28:51.530714 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.530699    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:28:51.535336 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.535281    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p"]
Jul 17 19:28:51.568658 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.568623    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e6fe30d-905c-439f-9793-88bca1af0e43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p\" (UID: \"1e6fe30d-905c-439f-9793-88bca1af0e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p"
Jul 17 19:28:51.669577 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.669538    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e6fe30d-905c-439f-9793-88bca1af0e43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p\" (UID: \"1e6fe30d-905c-439f-9793-88bca1af0e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p"
Jul 17 19:28:51.670173 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.670151    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e6fe30d-905c-439f-9793-88bca1af0e43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p\" (UID: \"1e6fe30d-905c-439f-9793-88bca1af0e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p"
Jul 17 19:28:51.840832 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:51.840809    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p"
Jul 17 19:28:52.180275 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:52.180247    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p"]
Jul 17 19:28:52.181490 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:28:52.181467    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6fe30d_905c_439f_9793_88bca1af0e43.slice/crio-08e5185e885572414732a2d17b3dfeb35c207b109fe32746ba0c32c3d60b98b2 WatchSource:0}: Error finding container 08e5185e885572414732a2d17b3dfeb35c207b109fe32746ba0c32c3d60b98b2: Status 404 returned error can't find the container with id 08e5185e885572414732a2d17b3dfeb35c207b109fe32746ba0c32c3d60b98b2
Jul 17 19:28:52.740685 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:52.740651    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p" event={"ID":"1e6fe30d-905c-439f-9793-88bca1af0e43","Type":"ContainerStarted","Data":"c774d743a38d37dd57b249a271e1cf05effe90ca68c9d295f3920dbd6ecfffc1"}
Jul 17 19:28:52.741065 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:52.740718    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p" event={"ID":"1e6fe30d-905c-439f-9793-88bca1af0e43","Type":"ContainerStarted","Data":"08e5185e885572414732a2d17b3dfeb35c207b109fe32746ba0c32c3d60b98b2"}
Jul 17 19:28:53.747784 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:53.747745    2559 generic.go:356] "Generic (PLEG): container finished" podID="1e6fe30d-905c-439f-9793-88bca1af0e43" containerID="c774d743a38d37dd57b249a271e1cf05effe90ca68c9d295f3920dbd6ecfffc1" exitCode=6
Jul 17 19:28:53.748194 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:53.747841    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p" event={"ID":"1e6fe30d-905c-439f-9793-88bca1af0e43","Type":"ContainerDied","Data":"c774d743a38d37dd57b249a271e1cf05effe90ca68c9d295f3920dbd6ecfffc1"}
Jul 17 19:28:53.748194 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:53.747950    2559 scope.go:117] "RemoveContainer" containerID="c774d743a38d37dd57b249a271e1cf05effe90ca68c9d295f3920dbd6ecfffc1"
Jul 17 19:28:54.755177 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:54.755106    2559 generic.go:356] "Generic (PLEG): container finished" podID="1e6fe30d-905c-439f-9793-88bca1af0e43" containerID="02f02d05bae780c70dbf135f70121c4a7a55c7e7a228660cf074bb3369cbdf70" exitCode=6
Jul 17 19:28:54.755563 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:54.755190    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p" event={"ID":"1e6fe30d-905c-439f-9793-88bca1af0e43","Type":"ContainerDied","Data":"02f02d05bae780c70dbf135f70121c4a7a55c7e7a228660cf074bb3369cbdf70"}
Jul 17 19:28:54.755563 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:54.755245    2559 scope.go:117] "RemoveContainer" containerID="c774d743a38d37dd57b249a271e1cf05effe90ca68c9d295f3920dbd6ecfffc1"
Jul 17 19:28:54.755563 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:54.755427    2559 scope.go:117] "RemoveContainer" containerID="02f02d05bae780c70dbf135f70121c4a7a55c7e7a228660cf074bb3369cbdf70"
Jul 17 19:28:54.755702 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:28:54.755645    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p_odh-ai-gateway-infra(1e6fe30d-905c-439f-9793-88bca1af0e43)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p" podUID="1e6fe30d-905c-439f-9793-88bca1af0e43"
Jul 17 19:28:55.763191 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:55.763161    2559 scope.go:117] "RemoveContainer" containerID="02f02d05bae780c70dbf135f70121c4a7a55c7e7a228660cf074bb3369cbdf70"
Jul 17 19:28:55.763621 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:28:55.763402    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p_odh-ai-gateway-infra(1e6fe30d-905c-439f-9793-88bca1af0e43)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p" podUID="1e6fe30d-905c-439f-9793-88bca1af0e43"
Jul 17 19:28:56.201144 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:56.201116    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm"]
Jul 17 19:28:56.201538 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:56.201524    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="882ca181-6d52-45a3-a989-fac488a2813a" containerName="revoke-keys"
Jul 17 19:28:56.201587 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:56.201541    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="882ca181-6d52-45a3-a989-fac488a2813a" containerName="revoke-keys"
Jul 17 19:28:56.201587 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:56.201553    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5f78e9b-0d57-4ae3-8e17-bd3e08216bda" containerName="revoke-keys"
Jul 17 19:28:56.201587 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:56.201558    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5f78e9b-0d57-4ae3-8e17-bd3e08216bda" containerName="revoke-keys"
Jul 17 19:28:56.201686 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:56.201632    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5f78e9b-0d57-4ae3-8e17-bd3e08216bda" containerName="revoke-keys"
Jul 17 19:28:56.205859 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:56.205841    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm"
Jul 17 19:28:56.211600 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:56.211546    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm"]
Jul 17 19:28:56.310060 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:56.310033    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e03fbc-b1f4-43c0-9fd0-0c33af90144e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm\" (UID: \"a1e03fbc-b1f4-43c0-9fd0-0c33af90144e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm"
Jul 17 19:28:56.411173 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:56.411148    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e03fbc-b1f4-43c0-9fd0-0c33af90144e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm\" (UID: \"a1e03fbc-b1f4-43c0-9fd0-0c33af90144e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm"
Jul 17 19:28:56.411750 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:56.411733    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e03fbc-b1f4-43c0-9fd0-0c33af90144e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm\" (UID: \"a1e03fbc-b1f4-43c0-9fd0-0c33af90144e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm"
Jul 17 19:28:56.517891 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:56.517820    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm"
Jul 17 19:28:56.657531 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:56.657506    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm"]
Jul 17 19:28:56.659012 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:28:56.658984    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e03fbc_b1f4_43c0_9fd0_0c33af90144e.slice/crio-b672cf29eafecf5f480e8c6383f8d8198bdb26cf97eb41157952dd5abbb31b90 WatchSource:0}: Error finding container b672cf29eafecf5f480e8c6383f8d8198bdb26cf97eb41157952dd5abbb31b90: Status 404 returned error can't find the container with id b672cf29eafecf5f480e8c6383f8d8198bdb26cf97eb41157952dd5abbb31b90
Jul 17 19:28:56.771342 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:56.771240    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm" event={"ID":"a1e03fbc-b1f4-43c0-9fd0-0c33af90144e","Type":"ContainerStarted","Data":"b672cf29eafecf5f480e8c6383f8d8198bdb26cf97eb41157952dd5abbb31b90"}
Jul 17 19:28:57.779174 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:57.779142    2559 generic.go:356] "Generic (PLEG): container finished" podID="a1e03fbc-b1f4-43c0-9fd0-0c33af90144e" containerID="762ce9187f0392f11fafd71cfcc51c6accf31e315b6cfd5f531a9dda950be2d4" exitCode=6
Jul 17 19:28:57.779623 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:57.779246    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm" event={"ID":"a1e03fbc-b1f4-43c0-9fd0-0c33af90144e","Type":"ContainerDied","Data":"762ce9187f0392f11fafd71cfcc51c6accf31e315b6cfd5f531a9dda950be2d4"}
Jul 17 19:28:57.779623 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:57.779465    2559 scope.go:117] "RemoveContainer" containerID="762ce9187f0392f11fafd71cfcc51c6accf31e315b6cfd5f531a9dda950be2d4"
Jul 17 19:28:58.787725 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:58.787639    2559 generic.go:356] "Generic (PLEG): container finished" podID="a1e03fbc-b1f4-43c0-9fd0-0c33af90144e" containerID="6bc112a4d6ee959013e1b31b926eb71e4bb00d6be92dc3aa3143fcb664e04cde" exitCode=6
Jul 17 19:28:58.787725 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:58.787678    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm" event={"ID":"a1e03fbc-b1f4-43c0-9fd0-0c33af90144e","Type":"ContainerDied","Data":"6bc112a4d6ee959013e1b31b926eb71e4bb00d6be92dc3aa3143fcb664e04cde"}
Jul 17 19:28:58.788216 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:58.787747    2559 scope.go:117] "RemoveContainer" containerID="762ce9187f0392f11fafd71cfcc51c6accf31e315b6cfd5f531a9dda950be2d4"
Jul 17 19:28:58.788216 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:58.787946    2559 scope.go:117] "RemoveContainer" containerID="6bc112a4d6ee959013e1b31b926eb71e4bb00d6be92dc3aa3143fcb664e04cde"
Jul 17 19:28:58.788216 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:28:58.788174    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm_odh-ai-gateway-infra(a1e03fbc-b1f4-43c0-9fd0-0c33af90144e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm" podUID="a1e03fbc-b1f4-43c0-9fd0-0c33af90144e"
Jul 17 19:28:59.801346 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:28:59.801289    2559 scope.go:117] "RemoveContainer" containerID="6bc112a4d6ee959013e1b31b926eb71e4bb00d6be92dc3aa3143fcb664e04cde"
Jul 17 19:28:59.801717 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:28:59.801533    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm_odh-ai-gateway-infra(a1e03fbc-b1f4-43c0-9fd0-0c33af90144e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm" podUID="a1e03fbc-b1f4-43c0-9fd0-0c33af90144e"
Jul 17 19:29:08.119856 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:08.119824    2559 scope.go:117] "RemoveContainer" containerID="02f02d05bae780c70dbf135f70121c4a7a55c7e7a228660cf074bb3369cbdf70"
Jul 17 19:29:08.859661 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:08.859623    2559 generic.go:356] "Generic (PLEG): container finished" podID="1e6fe30d-905c-439f-9793-88bca1af0e43" containerID="e6fb6aeb3b44c948cab5854d1285752d6082f10a0f453701e092e0e6055eeeba" exitCode=6
Jul 17 19:29:08.859790 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:08.859694    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p" event={"ID":"1e6fe30d-905c-439f-9793-88bca1af0e43","Type":"ContainerDied","Data":"e6fb6aeb3b44c948cab5854d1285752d6082f10a0f453701e092e0e6055eeeba"}
Jul 17 19:29:08.859790 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:08.859757    2559 scope.go:117] "RemoveContainer" containerID="02f02d05bae780c70dbf135f70121c4a7a55c7e7a228660cf074bb3369cbdf70"
Jul 17 19:29:08.860042 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:08.860024    2559 scope.go:117] "RemoveContainer" containerID="e6fb6aeb3b44c948cab5854d1285752d6082f10a0f453701e092e0e6055eeeba"
Jul 17 19:29:08.860277 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:29:08.860252    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p_odh-ai-gateway-infra(1e6fe30d-905c-439f-9793-88bca1af0e43)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p" podUID="1e6fe30d-905c-439f-9793-88bca1af0e43"
Jul 17 19:29:09.887415 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:09.887378    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p"]
Jul 17 19:29:10.025165 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:10.025136    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p"
Jul 17 19:29:10.131810 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:10.131784    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e6fe30d-905c-439f-9793-88bca1af0e43-maas-api-service-ca\") pod \"1e6fe30d-905c-439f-9793-88bca1af0e43\" (UID: \"1e6fe30d-905c-439f-9793-88bca1af0e43\") "
Jul 17 19:29:10.132130 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:10.132107    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e6fe30d-905c-439f-9793-88bca1af0e43-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1e6fe30d-905c-439f-9793-88bca1af0e43" (UID: "1e6fe30d-905c-439f-9793-88bca1af0e43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:29:10.232834 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:10.232748    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e6fe30d-905c-439f-9793-88bca1af0e43-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:29:10.875041 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:10.875006    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p" event={"ID":"1e6fe30d-905c-439f-9793-88bca1af0e43","Type":"ContainerDied","Data":"08e5185e885572414732a2d17b3dfeb35c207b109fe32746ba0c32c3d60b98b2"}
Jul 17 19:29:10.875041 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:10.875026    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p"
Jul 17 19:29:10.875276 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:10.875094    2559 scope.go:117] "RemoveContainer" containerID="e6fb6aeb3b44c948cab5854d1285752d6082f10a0f453701e092e0e6055eeeba"
Jul 17 19:29:10.898798 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:10.898769    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p"]
Jul 17 19:29:10.903927 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:10.903904    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-g8z2p"]
Jul 17 19:29:12.126054 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:12.126021    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e6fe30d-905c-439f-9793-88bca1af0e43" path="/var/lib/kubelet/pods/1e6fe30d-905c-439f-9793-88bca1af0e43/volumes"
Jul 17 19:29:15.120409 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:15.120383    2559 scope.go:117] "RemoveContainer" containerID="6bc112a4d6ee959013e1b31b926eb71e4bb00d6be92dc3aa3143fcb664e04cde"
Jul 17 19:29:15.907336 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:15.907292    2559 generic.go:356] "Generic (PLEG): container finished" podID="a1e03fbc-b1f4-43c0-9fd0-0c33af90144e" containerID="941eec3c0bdb5ec3ffe5b35e69f513ff97eb7f48f4b8114d027516a4d773a338" exitCode=6
Jul 17 19:29:15.907482 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:15.907372    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm" event={"ID":"a1e03fbc-b1f4-43c0-9fd0-0c33af90144e","Type":"ContainerDied","Data":"941eec3c0bdb5ec3ffe5b35e69f513ff97eb7f48f4b8114d027516a4d773a338"}
Jul 17 19:29:15.907482 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:15.907451    2559 scope.go:117] "RemoveContainer" containerID="6bc112a4d6ee959013e1b31b926eb71e4bb00d6be92dc3aa3143fcb664e04cde"
Jul 17 19:29:15.907733 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:15.907712    2559 scope.go:117] "RemoveContainer" containerID="941eec3c0bdb5ec3ffe5b35e69f513ff97eb7f48f4b8114d027516a4d773a338"
Jul 17 19:29:15.907918 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:29:15.907896    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm_odh-ai-gateway-infra(a1e03fbc-b1f4-43c0-9fd0-0c33af90144e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm" podUID="a1e03fbc-b1f4-43c0-9fd0-0c33af90144e"
Jul 17 19:29:16.933052 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:16.933018    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm"]
Jul 17 19:29:17.074615 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:17.074589    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm"
Jul 17 19:29:17.192092 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:17.192025    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e03fbc-b1f4-43c0-9fd0-0c33af90144e-maas-api-service-ca\") pod \"a1e03fbc-b1f4-43c0-9fd0-0c33af90144e\" (UID: \"a1e03fbc-b1f4-43c0-9fd0-0c33af90144e\") "
Jul 17 19:29:17.192285 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:17.192264    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e03fbc-b1f4-43c0-9fd0-0c33af90144e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a1e03fbc-b1f4-43c0-9fd0-0c33af90144e" (UID: "a1e03fbc-b1f4-43c0-9fd0-0c33af90144e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:29:17.293101 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:17.293074    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1e03fbc-b1f4-43c0-9fd0-0c33af90144e-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:29:17.923346 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:17.923299    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm"
Jul 17 19:29:17.923501 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:17.923298    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm" event={"ID":"a1e03fbc-b1f4-43c0-9fd0-0c33af90144e","Type":"ContainerDied","Data":"b672cf29eafecf5f480e8c6383f8d8198bdb26cf97eb41157952dd5abbb31b90"}
Jul 17 19:29:17.923501 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:17.923443    2559 scope.go:117] "RemoveContainer" containerID="941eec3c0bdb5ec3ffe5b35e69f513ff97eb7f48f4b8114d027516a4d773a338"
Jul 17 19:29:17.945001 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:17.944977    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm"]
Jul 17 19:29:17.950484 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:17.950465    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-tphqm"]
Jul 17 19:29:18.126379 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:18.126349    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1e03fbc-b1f4-43c0-9fd0-0c33af90144e" path="/var/lib/kubelet/pods/a1e03fbc-b1f4-43c0-9fd0-0c33af90144e/volumes"
Jul 17 19:29:23.301324 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.301275    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp"]
Jul 17 19:29:23.302151 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.302128    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1e03fbc-b1f4-43c0-9fd0-0c33af90144e" containerName="revoke-keys"
Jul 17 19:29:23.302256 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.302155    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1e03fbc-b1f4-43c0-9fd0-0c33af90144e" containerName="revoke-keys"
Jul 17 19:29:23.302256 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.302176    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1e03fbc-b1f4-43c0-9fd0-0c33af90144e" containerName="revoke-keys"
Jul 17 19:29:23.302256 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.302184    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1e03fbc-b1f4-43c0-9fd0-0c33af90144e" containerName="revoke-keys"
Jul 17 19:29:23.302256 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.302249    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e6fe30d-905c-439f-9793-88bca1af0e43" containerName="revoke-keys"
Jul 17 19:29:23.302490 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.302264    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e6fe30d-905c-439f-9793-88bca1af0e43" containerName="revoke-keys"
Jul 17 19:29:23.302490 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.302277    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e6fe30d-905c-439f-9793-88bca1af0e43" containerName="revoke-keys"
Jul 17 19:29:23.302490 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.302290    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e6fe30d-905c-439f-9793-88bca1af0e43" containerName="revoke-keys"
Jul 17 19:29:23.302639 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.302510    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1e03fbc-b1f4-43c0-9fd0-0c33af90144e" containerName="revoke-keys"
Jul 17 19:29:23.302639 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.302532    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e6fe30d-905c-439f-9793-88bca1af0e43" containerName="revoke-keys"
Jul 17 19:29:23.302639 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.302540    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e6fe30d-905c-439f-9793-88bca1af0e43" containerName="revoke-keys"
Jul 17 19:29:23.302639 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.302557    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e6fe30d-905c-439f-9793-88bca1af0e43" containerName="revoke-keys"
Jul 17 19:29:23.302639 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.302564    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1e03fbc-b1f4-43c0-9fd0-0c33af90144e" containerName="revoke-keys"
Jul 17 19:29:23.302639 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.302579    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1e03fbc-b1f4-43c0-9fd0-0c33af90144e" containerName="revoke-keys"
Jul 17 19:29:23.306813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.306779    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp"
Jul 17 19:29:23.308957 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.308929    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp"]
Jul 17 19:29:23.309259 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.309235    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:29:23.438377 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.438339    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c8b866c-c89c-4566-a62b-e7d091a6da97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp\" (UID: \"8c8b866c-c89c-4566-a62b-e7d091a6da97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp"
Jul 17 19:29:23.539711 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.539683    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c8b866c-c89c-4566-a62b-e7d091a6da97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp\" (UID: \"8c8b866c-c89c-4566-a62b-e7d091a6da97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp"
Jul 17 19:29:23.540355 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.540335    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c8b866c-c89c-4566-a62b-e7d091a6da97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp\" (UID: \"8c8b866c-c89c-4566-a62b-e7d091a6da97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp"
Jul 17 19:29:23.619284 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.619263    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp"
Jul 17 19:29:23.752070 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.752045    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp"]
Jul 17 19:29:23.753477 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:29:23.753454    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8b866c_c89c_4566_a62b_e7d091a6da97.slice/crio-485bd6f4829e3852bd4afb2e058c38bc90f2359cf9248a728fabb9d7f1db87c8 WatchSource:0}: Error finding container 485bd6f4829e3852bd4afb2e058c38bc90f2359cf9248a728fabb9d7f1db87c8: Status 404 returned error can't find the container with id 485bd6f4829e3852bd4afb2e058c38bc90f2359cf9248a728fabb9d7f1db87c8
Jul 17 19:29:23.960140 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:23.960108    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp" event={"ID":"8c8b866c-c89c-4566-a62b-e7d091a6da97","Type":"ContainerStarted","Data":"485bd6f4829e3852bd4afb2e058c38bc90f2359cf9248a728fabb9d7f1db87c8"}
Jul 17 19:29:24.967770 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:24.967735    2559 generic.go:356] "Generic (PLEG): container finished" podID="8c8b866c-c89c-4566-a62b-e7d091a6da97" containerID="76711518759bb759ae83b6f2250a1861f2120ade73195d67132c3c7cfd76ff50" exitCode=6
Jul 17 19:29:24.968258 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:24.967824    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp" event={"ID":"8c8b866c-c89c-4566-a62b-e7d091a6da97","Type":"ContainerDied","Data":"76711518759bb759ae83b6f2250a1861f2120ade73195d67132c3c7cfd76ff50"}
Jul 17 19:29:24.968258 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:24.967984    2559 scope.go:117] "RemoveContainer" containerID="76711518759bb759ae83b6f2250a1861f2120ade73195d67132c3c7cfd76ff50"
Jul 17 19:29:25.975245 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:25.975211    2559 generic.go:356] "Generic (PLEG): container finished" podID="8c8b866c-c89c-4566-a62b-e7d091a6da97" containerID="45dfbaff9e5afccfc573d63ed99136260a19b2363eaf9782668265885762284f" exitCode=6
Jul 17 19:29:25.975691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:25.975300    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp" event={"ID":"8c8b866c-c89c-4566-a62b-e7d091a6da97","Type":"ContainerDied","Data":"45dfbaff9e5afccfc573d63ed99136260a19b2363eaf9782668265885762284f"}
Jul 17 19:29:25.975691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:25.975382    2559 scope.go:117] "RemoveContainer" containerID="76711518759bb759ae83b6f2250a1861f2120ade73195d67132c3c7cfd76ff50"
Jul 17 19:29:25.975691 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:25.975579    2559 scope.go:117] "RemoveContainer" containerID="45dfbaff9e5afccfc573d63ed99136260a19b2363eaf9782668265885762284f"
Jul 17 19:29:25.975825 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:29:25.975806    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp_odh-ai-gateway-infra(8c8b866c-c89c-4566-a62b-e7d091a6da97)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp" podUID="8c8b866c-c89c-4566-a62b-e7d091a6da97"
Jul 17 19:29:26.983826 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:26.983800    2559 scope.go:117] "RemoveContainer" containerID="45dfbaff9e5afccfc573d63ed99136260a19b2363eaf9782668265885762284f"
Jul 17 19:29:26.984251 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:29:26.984037    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp_odh-ai-gateway-infra(8c8b866c-c89c-4566-a62b-e7d091a6da97)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp" podUID="8c8b866c-c89c-4566-a62b-e7d091a6da97"
Jul 17 19:29:39.119898 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:39.119866    2559 scope.go:117] "RemoveContainer" containerID="45dfbaff9e5afccfc573d63ed99136260a19b2363eaf9782668265885762284f"
Jul 17 19:29:40.063703 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:40.063669    2559 generic.go:356] "Generic (PLEG): container finished" podID="8c8b866c-c89c-4566-a62b-e7d091a6da97" containerID="9baddeeeeac730d4b32eec5f8036c10506168221ec7f19084a16b159d59e7bd5" exitCode=6
Jul 17 19:29:40.063878 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:40.063745    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp" event={"ID":"8c8b866c-c89c-4566-a62b-e7d091a6da97","Type":"ContainerDied","Data":"9baddeeeeac730d4b32eec5f8036c10506168221ec7f19084a16b159d59e7bd5"}
Jul 17 19:29:40.063878 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:40.063815    2559 scope.go:117] "RemoveContainer" containerID="45dfbaff9e5afccfc573d63ed99136260a19b2363eaf9782668265885762284f"
Jul 17 19:29:40.064155 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:40.064131    2559 scope.go:117] "RemoveContainer" containerID="9baddeeeeac730d4b32eec5f8036c10506168221ec7f19084a16b159d59e7bd5"
Jul 17 19:29:40.064434 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:29:40.064398    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp_odh-ai-gateway-infra(8c8b866c-c89c-4566-a62b-e7d091a6da97)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp" podUID="8c8b866c-c89c-4566-a62b-e7d091a6da97"
Jul 17 19:29:41.097809 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:41.097777    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp"]
Jul 17 19:29:41.233340 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:41.233301    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp"
Jul 17 19:29:41.401485 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:41.401397    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c8b866c-c89c-4566-a62b-e7d091a6da97-maas-api-service-ca\") pod \"8c8b866c-c89c-4566-a62b-e7d091a6da97\" (UID: \"8c8b866c-c89c-4566-a62b-e7d091a6da97\") "
Jul 17 19:29:41.401775 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:41.401750    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8b866c-c89c-4566-a62b-e7d091a6da97-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c8b866c-c89c-4566-a62b-e7d091a6da97" (UID: "8c8b866c-c89c-4566-a62b-e7d091a6da97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:29:41.502496 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:41.502469    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c8b866c-c89c-4566-a62b-e7d091a6da97-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:29:42.079388 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:42.079352    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp"
Jul 17 19:29:42.079589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:42.079349    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp" event={"ID":"8c8b866c-c89c-4566-a62b-e7d091a6da97","Type":"ContainerDied","Data":"485bd6f4829e3852bd4afb2e058c38bc90f2359cf9248a728fabb9d7f1db87c8"}
Jul 17 19:29:42.079589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:42.079535    2559 scope.go:117] "RemoveContainer" containerID="9baddeeeeac730d4b32eec5f8036c10506168221ec7f19084a16b159d59e7bd5"
Jul 17 19:29:42.100392 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:42.100367    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp"]
Jul 17 19:29:42.104628 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:42.104604    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7dlxp"]
Jul 17 19:29:42.125570 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:42.125547    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c8b866c-c89c-4566-a62b-e7d091a6da97" path="/var/lib/kubelet/pods/8c8b866c-c89c-4566-a62b-e7d091a6da97/volumes"
Jul 17 19:29:45.864828 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:45.864794    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"]
Jul 17 19:29:46.711395 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:46.711358    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"]
Jul 17 19:29:56.251912 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.251880    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97"]
Jul 17 19:29:56.252456 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.252252    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c8b866c-c89c-4566-a62b-e7d091a6da97" containerName="revoke-keys"
Jul 17 19:29:56.252456 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.252263    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c8b866c-c89c-4566-a62b-e7d091a6da97" containerName="revoke-keys"
Jul 17 19:29:56.252456 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.252279    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e6fe30d-905c-439f-9793-88bca1af0e43" containerName="revoke-keys"
Jul 17 19:29:56.252456 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.252285    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e6fe30d-905c-439f-9793-88bca1af0e43" containerName="revoke-keys"
Jul 17 19:29:56.252456 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.252291    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c8b866c-c89c-4566-a62b-e7d091a6da97" containerName="revoke-keys"
Jul 17 19:29:56.252456 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.252297    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c8b866c-c89c-4566-a62b-e7d091a6da97" containerName="revoke-keys"
Jul 17 19:29:56.252456 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.252322    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1e03fbc-b1f4-43c0-9fd0-0c33af90144e" containerName="revoke-keys"
Jul 17 19:29:56.252456 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.252328    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1e03fbc-b1f4-43c0-9fd0-0c33af90144e" containerName="revoke-keys"
Jul 17 19:29:56.252456 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.252334    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c8b866c-c89c-4566-a62b-e7d091a6da97" containerName="revoke-keys"
Jul 17 19:29:56.252456 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.252339    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c8b866c-c89c-4566-a62b-e7d091a6da97" containerName="revoke-keys"
Jul 17 19:29:56.252456 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.252401    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c8b866c-c89c-4566-a62b-e7d091a6da97" containerName="revoke-keys"
Jul 17 19:29:56.252456 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.252409    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c8b866c-c89c-4566-a62b-e7d091a6da97" containerName="revoke-keys"
Jul 17 19:29:56.252456 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.252417    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c8b866c-c89c-4566-a62b-e7d091a6da97" containerName="revoke-keys"
Jul 17 19:29:56.255556 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.255540    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97"
Jul 17 19:29:56.258069 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.258039    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:29:56.262803 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.262781    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97"]
Jul 17 19:29:56.329087 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.329050    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37f03155-aeed-4ff1-848e-4b4702c06fa7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97\" (UID: \"37f03155-aeed-4ff1-848e-4b4702c06fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97"
Jul 17 19:29:56.429799 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.429766    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37f03155-aeed-4ff1-848e-4b4702c06fa7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97\" (UID: \"37f03155-aeed-4ff1-848e-4b4702c06fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97"
Jul 17 19:29:56.430412 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.430392    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37f03155-aeed-4ff1-848e-4b4702c06fa7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97\" (UID: \"37f03155-aeed-4ff1-848e-4b4702c06fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97"
Jul 17 19:29:56.567781 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.567751    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97"
Jul 17 19:29:56.706993 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:56.706961    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97"]
Jul 17 19:29:56.708611 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:29:56.708581    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f03155_aeed_4ff1_848e_4b4702c06fa7.slice/crio-b503b453b176934a1418736bb2b4d354351d872b6c665190ac9cfc76c69f5770 WatchSource:0}: Error finding container b503b453b176934a1418736bb2b4d354351d872b6c665190ac9cfc76c69f5770: Status 404 returned error can't find the container with id b503b453b176934a1418736bb2b4d354351d872b6c665190ac9cfc76c69f5770
Jul 17 19:29:57.172694 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:57.172586    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97" event={"ID":"37f03155-aeed-4ff1-848e-4b4702c06fa7","Type":"ContainerStarted","Data":"3b65562c9fcec2e892afa2590ab5a3bfee3a4d0b6ed3ca9fd037de2ce235c29f"}
Jul 17 19:29:57.172694 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:57.172647    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97" event={"ID":"37f03155-aeed-4ff1-848e-4b4702c06fa7","Type":"ContainerStarted","Data":"b503b453b176934a1418736bb2b4d354351d872b6c665190ac9cfc76c69f5770"}
Jul 17 19:29:57.188300 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:57.188254    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97" podStartSLOduration=1.188238007 podStartE2EDuration="1.188238007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:29:57.18721361 +0000 UTC m=+3407.628722300" watchObservedRunningTime="2026-07-17 19:29:57.188238007 +0000 UTC m=+3407.629746696"
Jul 17 19:29:58.179969 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:58.179935    2559 generic.go:356] "Generic (PLEG): container finished" podID="37f03155-aeed-4ff1-848e-4b4702c06fa7" containerID="3b65562c9fcec2e892afa2590ab5a3bfee3a4d0b6ed3ca9fd037de2ce235c29f" exitCode=6
Jul 17 19:29:58.180496 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:58.180022    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97" event={"ID":"37f03155-aeed-4ff1-848e-4b4702c06fa7","Type":"ContainerDied","Data":"3b65562c9fcec2e892afa2590ab5a3bfee3a4d0b6ed3ca9fd037de2ce235c29f"}
Jul 17 19:29:58.180496 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:58.180222    2559 scope.go:117] "RemoveContainer" containerID="3b65562c9fcec2e892afa2590ab5a3bfee3a4d0b6ed3ca9fd037de2ce235c29f"
Jul 17 19:29:59.190414 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:59.190366    2559 generic.go:356] "Generic (PLEG): container finished" podID="37f03155-aeed-4ff1-848e-4b4702c06fa7" containerID="74063ff34927d5ac5702f9ca62e1d59f3ec1fed82b0baec764adfd04d1740d88" exitCode=6
Jul 17 19:29:59.190880 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:59.190430    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97" event={"ID":"37f03155-aeed-4ff1-848e-4b4702c06fa7","Type":"ContainerDied","Data":"74063ff34927d5ac5702f9ca62e1d59f3ec1fed82b0baec764adfd04d1740d88"}
Jul 17 19:29:59.190880 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:59.190488    2559 scope.go:117] "RemoveContainer" containerID="3b65562c9fcec2e892afa2590ab5a3bfee3a4d0b6ed3ca9fd037de2ce235c29f"
Jul 17 19:29:59.190880 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:29:59.190672    2559 scope.go:117] "RemoveContainer" containerID="74063ff34927d5ac5702f9ca62e1d59f3ec1fed82b0baec764adfd04d1740d88"
Jul 17 19:29:59.191021 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:29:59.190933    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97_odh-ai-gateway-infra(37f03155-aeed-4ff1-848e-4b4702c06fa7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97" podUID="37f03155-aeed-4ff1-848e-4b4702c06fa7"
Jul 17 19:30:00.197950 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:00.197876    2559 scope.go:117] "RemoveContainer" containerID="74063ff34927d5ac5702f9ca62e1d59f3ec1fed82b0baec764adfd04d1740d88"
Jul 17 19:30:00.198325 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:30:00.198107    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97_odh-ai-gateway-infra(37f03155-aeed-4ff1-848e-4b4702c06fa7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97" podUID="37f03155-aeed-4ff1-848e-4b4702c06fa7"
Jul 17 19:30:00.890753 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:00.890708    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2" podUID="d2f5f38a-2de7-4773-b4d5-66fcb47662e1" containerName="main" containerID="cri-o://df59e2e8bb289ca61270617ce2ad9379e912c4eacaa13dd06726ebcd0658bec0" gracePeriod=45
Jul 17 19:30:01.007349 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.007285    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2" podUID="d2f5f38a-2de7-4773-b4d5-66fcb47662e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": dial tcp 10.133.0.115:8000: connect: connection refused"
Jul 17 19:30:01.147194 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.147128    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:30:01.205516 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.205484    2559 generic.go:356] "Generic (PLEG): container finished" podID="d2f5f38a-2de7-4773-b4d5-66fcb47662e1" containerID="df59e2e8bb289ca61270617ce2ad9379e912c4eacaa13dd06726ebcd0658bec0" exitCode=0
Jul 17 19:30:01.205994 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.205527    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"
Jul 17 19:30:01.205994 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.205575    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2" event={"ID":"d2f5f38a-2de7-4773-b4d5-66fcb47662e1","Type":"ContainerDied","Data":"df59e2e8bb289ca61270617ce2ad9379e912c4eacaa13dd06726ebcd0658bec0"}
Jul 17 19:30:01.205994 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.205624    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2" event={"ID":"d2f5f38a-2de7-4773-b4d5-66fcb47662e1","Type":"ContainerDied","Data":"b98c22f62f06a5a6e78ea6bc6c773f7d8a842fe59a31c856a7781108e70661b1"}
Jul 17 19:30:01.205994 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.205651    2559 scope.go:117] "RemoveContainer" containerID="df59e2e8bb289ca61270617ce2ad9379e912c4eacaa13dd06726ebcd0658bec0"
Jul 17 19:30:01.215495 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.215446    2559 scope.go:117] "RemoveContainer" containerID="df59e2e8bb289ca61270617ce2ad9379e912c4eacaa13dd06726ebcd0658bec0"
Jul 17 19:30:01.215779 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:30:01.215752    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df59e2e8bb289ca61270617ce2ad9379e912c4eacaa13dd06726ebcd0658bec0\": container with ID starting with df59e2e8bb289ca61270617ce2ad9379e912c4eacaa13dd06726ebcd0658bec0 not found: ID does not exist" containerID="df59e2e8bb289ca61270617ce2ad9379e912c4eacaa13dd06726ebcd0658bec0"
Jul 17 19:30:01.215849 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.215795    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df59e2e8bb289ca61270617ce2ad9379e912c4eacaa13dd06726ebcd0658bec0"} err="failed to get container status \"df59e2e8bb289ca61270617ce2ad9379e912c4eacaa13dd06726ebcd0658bec0\": rpc error: code = NotFound desc = could not find container \"df59e2e8bb289ca61270617ce2ad9379e912c4eacaa13dd06726ebcd0658bec0\": container with ID starting with df59e2e8bb289ca61270617ce2ad9379e912c4eacaa13dd06726ebcd0658bec0 not found: ID does not exist"
Jul 17 19:30:01.276405 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.276371    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lqwk\" (UniqueName: \"kubernetes.io/projected/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-kube-api-access-9lqwk\") pod \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") "
Jul 17 19:30:01.276561 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.276441    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-home\") pod \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") "
Jul 17 19:30:01.276561 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.276475    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-tmp-dir\") pod \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") "
Jul 17 19:30:01.276561 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.276516    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-dshm\") pod \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") "
Jul 17 19:30:01.276734 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.276597    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-model-cache\") pod \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") "
Jul 17 19:30:01.276734 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.276666    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-tls-certs\") pod \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\" (UID: \"d2f5f38a-2de7-4773-b4d5-66fcb47662e1\") "
Jul 17 19:30:01.276833 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.276745    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2f5f38a-2de7-4773-b4d5-66fcb47662e1" (UID: "d2f5f38a-2de7-4773-b4d5-66fcb47662e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:30:01.276833 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.276770    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-home" (OuterVolumeSpecName: "home") pod "d2f5f38a-2de7-4773-b4d5-66fcb47662e1" (UID: "d2f5f38a-2de7-4773-b4d5-66fcb47662e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:30:01.277022 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.276971    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2f5f38a-2de7-4773-b4d5-66fcb47662e1" (UID: "d2f5f38a-2de7-4773-b4d5-66fcb47662e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:30:01.277369 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.277272    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-home\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:30:01.277369 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.277298    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-tmp-dir\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:30:01.277369 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.277344    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-model-cache\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:30:01.278787 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.278761    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-kube-api-access-9lqwk" (OuterVolumeSpecName: "kube-api-access-9lqwk") pod "d2f5f38a-2de7-4773-b4d5-66fcb47662e1" (UID: "d2f5f38a-2de7-4773-b4d5-66fcb47662e1"). InnerVolumeSpecName "kube-api-access-9lqwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:30:01.278887 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.278866    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2f5f38a-2de7-4773-b4d5-66fcb47662e1" (UID: "d2f5f38a-2de7-4773-b4d5-66fcb47662e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:30:01.278931 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.278878    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-dshm" (OuterVolumeSpecName: "dshm") pod "d2f5f38a-2de7-4773-b4d5-66fcb47662e1" (UID: "d2f5f38a-2de7-4773-b4d5-66fcb47662e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:30:01.378536 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.378502    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-tls-certs\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:30:01.378536 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.378533    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lqwk\" (UniqueName: \"kubernetes.io/projected/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-kube-api-access-9lqwk\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:30:01.378709 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.378546    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f5f38a-2de7-4773-b4d5-66fcb47662e1-dshm\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:30:01.528815 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.528787    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"]
Jul 17 19:30:01.534357 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.534332    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-98e305/auth-test-model-98e305-kserve-fc69f58f9-g5mg2"]
Jul 17 19:30:01.732593 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.732539    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86" podUID="47c1f092-722e-43d5-93aa-f6d5a64572f3" containerName="main" containerID="cri-o://0ac255dbea50b12c53bf5bfdeb63260fbecccfca23ffab17ffe2a9787b1d3577" gracePeriod=45
Jul 17 19:30:01.980984 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:01.980958    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:30:02.085016 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.084969    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-tmp-dir\") pod \"47c1f092-722e-43d5-93aa-f6d5a64572f3\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") "
Jul 17 19:30:02.085188 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.085070    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47c1f092-722e-43d5-93aa-f6d5a64572f3-tls-certs\") pod \"47c1f092-722e-43d5-93aa-f6d5a64572f3\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") "
Jul 17 19:30:02.085188 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.085106    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-model-cache\") pod \"47c1f092-722e-43d5-93aa-f6d5a64572f3\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") "
Jul 17 19:30:02.085188 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.085149    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrxl2\" (UniqueName: \"kubernetes.io/projected/47c1f092-722e-43d5-93aa-f6d5a64572f3-kube-api-access-jrxl2\") pod \"47c1f092-722e-43d5-93aa-f6d5a64572f3\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") "
Jul 17 19:30:02.085188 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.085185    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-home\") pod \"47c1f092-722e-43d5-93aa-f6d5a64572f3\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") "
Jul 17 19:30:02.085477 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.085232    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-dshm\") pod \"47c1f092-722e-43d5-93aa-f6d5a64572f3\" (UID: \"47c1f092-722e-43d5-93aa-f6d5a64572f3\") "
Jul 17 19:30:02.085477 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.085253    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "47c1f092-722e-43d5-93aa-f6d5a64572f3" (UID: "47c1f092-722e-43d5-93aa-f6d5a64572f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:30:02.085477 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.085373    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "47c1f092-722e-43d5-93aa-f6d5a64572f3" (UID: "47c1f092-722e-43d5-93aa-f6d5a64572f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:30:02.085629 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.085509    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-home" (OuterVolumeSpecName: "home") pod "47c1f092-722e-43d5-93aa-f6d5a64572f3" (UID: "47c1f092-722e-43d5-93aa-f6d5a64572f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:30:02.085629 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.085599    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-model-cache\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:30:02.085629 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.085616    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-home\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:30:02.085742 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.085630    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-tmp-dir\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:30:02.087471 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.087449    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47c1f092-722e-43d5-93aa-f6d5a64572f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47c1f092-722e-43d5-93aa-f6d5a64572f3" (UID: "47c1f092-722e-43d5-93aa-f6d5a64572f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:30:02.087576 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.087497    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-dshm" (OuterVolumeSpecName: "dshm") pod "47c1f092-722e-43d5-93aa-f6d5a64572f3" (UID: "47c1f092-722e-43d5-93aa-f6d5a64572f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:30:02.087576 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.087531    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c1f092-722e-43d5-93aa-f6d5a64572f3-kube-api-access-jrxl2" (OuterVolumeSpecName: "kube-api-access-jrxl2") pod "47c1f092-722e-43d5-93aa-f6d5a64572f3" (UID: "47c1f092-722e-43d5-93aa-f6d5a64572f3"). InnerVolumeSpecName "kube-api-access-jrxl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:30:02.126156 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.126092    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2f5f38a-2de7-4773-b4d5-66fcb47662e1" path="/var/lib/kubelet/pods/d2f5f38a-2de7-4773-b4d5-66fcb47662e1/volumes"
Jul 17 19:30:02.186546 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.186495    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47c1f092-722e-43d5-93aa-f6d5a64572f3-tls-certs\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:30:02.186546 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.186537    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrxl2\" (UniqueName: \"kubernetes.io/projected/47c1f092-722e-43d5-93aa-f6d5a64572f3-kube-api-access-jrxl2\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:30:02.186546 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.186549    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47c1f092-722e-43d5-93aa-f6d5a64572f3-dshm\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:30:02.213449 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.213414    2559 generic.go:356] "Generic (PLEG): container finished" podID="47c1f092-722e-43d5-93aa-f6d5a64572f3" containerID="0ac255dbea50b12c53bf5bfdeb63260fbecccfca23ffab17ffe2a9787b1d3577" exitCode=0
Jul 17 19:30:02.213859 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.213462    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"
Jul 17 19:30:02.213859 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.213508    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86" event={"ID":"47c1f092-722e-43d5-93aa-f6d5a64572f3","Type":"ContainerDied","Data":"0ac255dbea50b12c53bf5bfdeb63260fbecccfca23ffab17ffe2a9787b1d3577"}
Jul 17 19:30:02.213859 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.213553    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86" event={"ID":"47c1f092-722e-43d5-93aa-f6d5a64572f3","Type":"ContainerDied","Data":"6d50253a37d88de94dbaa9df2f70e60e48a145bb45459177b5d074ca2371fb74"}
Jul 17 19:30:02.213859 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.213571    2559 scope.go:117] "RemoveContainer" containerID="0ac255dbea50b12c53bf5bfdeb63260fbecccfca23ffab17ffe2a9787b1d3577"
Jul 17 19:30:02.222777 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.222759    2559 scope.go:117] "RemoveContainer" containerID="0ac255dbea50b12c53bf5bfdeb63260fbecccfca23ffab17ffe2a9787b1d3577"
Jul 17 19:30:02.223029 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:30:02.223008    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac255dbea50b12c53bf5bfdeb63260fbecccfca23ffab17ffe2a9787b1d3577\": container with ID starting with 0ac255dbea50b12c53bf5bfdeb63260fbecccfca23ffab17ffe2a9787b1d3577 not found: ID does not exist" containerID="0ac255dbea50b12c53bf5bfdeb63260fbecccfca23ffab17ffe2a9787b1d3577"
Jul 17 19:30:02.223080 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.223037    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac255dbea50b12c53bf5bfdeb63260fbecccfca23ffab17ffe2a9787b1d3577"} err="failed to get container status \"0ac255dbea50b12c53bf5bfdeb63260fbecccfca23ffab17ffe2a9787b1d3577\": rpc error: code = NotFound desc = could not find container \"0ac255dbea50b12c53bf5bfdeb63260fbecccfca23ffab17ffe2a9787b1d3577\": container with ID starting with 0ac255dbea50b12c53bf5bfdeb63260fbecccfca23ffab17ffe2a9787b1d3577 not found: ID does not exist"
Jul 17 19:30:02.230618 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.230596    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"]
Jul 17 19:30:02.234810 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:02.234790    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-2109fd/auth-test-model-2109fd-kserve-6ccb948499-2bx86"]
Jul 17 19:30:04.127723 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:04.127687    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47c1f092-722e-43d5-93aa-f6d5a64572f3" path="/var/lib/kubelet/pods/47c1f092-722e-43d5-93aa-f6d5a64572f3/volumes"
Jul 17 19:30:04.454336 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:04.454234    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265"]
Jul 17 19:30:04.454668 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:04.454655    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47c1f092-722e-43d5-93aa-f6d5a64572f3" containerName="main"
Jul 17 19:30:04.454711 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:04.454669    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="47c1f092-722e-43d5-93aa-f6d5a64572f3" containerName="main"
Jul 17 19:30:04.454711 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:04.454684    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f5f38a-2de7-4773-b4d5-66fcb47662e1" containerName="main"
Jul 17 19:30:04.454711 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:04.454690    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f5f38a-2de7-4773-b4d5-66fcb47662e1" containerName="main"
Jul 17 19:30:04.454834 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:04.454763    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f5f38a-2de7-4773-b4d5-66fcb47662e1" containerName="main"
Jul 17 19:30:04.454834 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:04.454773    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="47c1f092-722e-43d5-93aa-f6d5a64572f3" containerName="main"
Jul 17 19:30:04.459663 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:04.459646    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265"
Jul 17 19:30:04.465145 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:04.465115    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265"]
Jul 17 19:30:04.506688 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:04.506658    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa27fa0-d5e0-485c-8406-81d04e99044a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265\" (UID: \"faa27fa0-d5e0-485c-8406-81d04e99044a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265"
Jul 17 19:30:04.607145 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:04.607115    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa27fa0-d5e0-485c-8406-81d04e99044a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265\" (UID: \"faa27fa0-d5e0-485c-8406-81d04e99044a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265"
Jul 17 19:30:04.607745 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:04.607727    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa27fa0-d5e0-485c-8406-81d04e99044a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265\" (UID: \"faa27fa0-d5e0-485c-8406-81d04e99044a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265"
Jul 17 19:30:04.772168 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:04.772087    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265"
Jul 17 19:30:04.908637 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:04.908603    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265"]
Jul 17 19:30:04.910231 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:30:04.910205    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa27fa0_d5e0_485c_8406_81d04e99044a.slice/crio-ca318e82682acd2ce81dc628555b591b67f4cc7789244b4fb77366cdccbd35f2 WatchSource:0}: Error finding container ca318e82682acd2ce81dc628555b591b67f4cc7789244b4fb77366cdccbd35f2: Status 404 returned error can't find the container with id ca318e82682acd2ce81dc628555b591b67f4cc7789244b4fb77366cdccbd35f2
Jul 17 19:30:05.234826 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:05.234791    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265" event={"ID":"faa27fa0-d5e0-485c-8406-81d04e99044a","Type":"ContainerStarted","Data":"ca318e82682acd2ce81dc628555b591b67f4cc7789244b4fb77366cdccbd35f2"}
Jul 17 19:30:06.241940 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:06.241899    2559 generic.go:356] "Generic (PLEG): container finished" podID="faa27fa0-d5e0-485c-8406-81d04e99044a" containerID="4fb71ea82927a74c67c2f7a6b001242abc31f89dfaaeba89d562b9dab65d8ac1" exitCode=6
Jul 17 19:30:06.242395 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:06.241988    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265" event={"ID":"faa27fa0-d5e0-485c-8406-81d04e99044a","Type":"ContainerDied","Data":"4fb71ea82927a74c67c2f7a6b001242abc31f89dfaaeba89d562b9dab65d8ac1"}
Jul 17 19:30:06.242395 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:06.242135    2559 scope.go:117] "RemoveContainer" containerID="4fb71ea82927a74c67c2f7a6b001242abc31f89dfaaeba89d562b9dab65d8ac1"
Jul 17 19:30:07.249571 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:07.249490    2559 generic.go:356] "Generic (PLEG): container finished" podID="faa27fa0-d5e0-485c-8406-81d04e99044a" containerID="ffceb2dffe99eba7d99fa409d78470a0a6cf097bd2ec99121599b5be23689161" exitCode=6
Jul 17 19:30:07.249571 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:07.249534    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265" event={"ID":"faa27fa0-d5e0-485c-8406-81d04e99044a","Type":"ContainerDied","Data":"ffceb2dffe99eba7d99fa409d78470a0a6cf097bd2ec99121599b5be23689161"}
Jul 17 19:30:07.250060 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:07.249582    2559 scope.go:117] "RemoveContainer" containerID="4fb71ea82927a74c67c2f7a6b001242abc31f89dfaaeba89d562b9dab65d8ac1"
Jul 17 19:30:07.250060 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:07.249743    2559 scope.go:117] "RemoveContainer" containerID="ffceb2dffe99eba7d99fa409d78470a0a6cf097bd2ec99121599b5be23689161"
Jul 17 19:30:07.250060 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:30:07.249973    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265_odh-ai-gateway-infra(faa27fa0-d5e0-485c-8406-81d04e99044a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265" podUID="faa27fa0-d5e0-485c-8406-81d04e99044a"
Jul 17 19:30:08.260188 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:08.260154    2559 scope.go:117] "RemoveContainer" containerID="ffceb2dffe99eba7d99fa409d78470a0a6cf097bd2ec99121599b5be23689161"
Jul 17 19:30:08.260687 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:30:08.260431    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265_odh-ai-gateway-infra(faa27fa0-d5e0-485c-8406-81d04e99044a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265" podUID="faa27fa0-d5e0-485c-8406-81d04e99044a"
Jul 17 19:30:11.120366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:11.120338    2559 scope.go:117] "RemoveContainer" containerID="74063ff34927d5ac5702f9ca62e1d59f3ec1fed82b0baec764adfd04d1740d88"
Jul 17 19:30:12.285165 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:12.285127    2559 generic.go:356] "Generic (PLEG): container finished" podID="37f03155-aeed-4ff1-848e-4b4702c06fa7" containerID="f57caf6ab86a6bf9f000964a4b94a28b1e24ae0dd14ba0d16874039823f10cc0" exitCode=6
Jul 17 19:30:12.285650 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:12.285208    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97" event={"ID":"37f03155-aeed-4ff1-848e-4b4702c06fa7","Type":"ContainerDied","Data":"f57caf6ab86a6bf9f000964a4b94a28b1e24ae0dd14ba0d16874039823f10cc0"}
Jul 17 19:30:12.285650 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:12.285267    2559 scope.go:117] "RemoveContainer" containerID="74063ff34927d5ac5702f9ca62e1d59f3ec1fed82b0baec764adfd04d1740d88"
Jul 17 19:30:12.285650 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:12.285555    2559 scope.go:117] "RemoveContainer" containerID="f57caf6ab86a6bf9f000964a4b94a28b1e24ae0dd14ba0d16874039823f10cc0"
Jul 17 19:30:12.285811 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:30:12.285751    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97_odh-ai-gateway-infra(37f03155-aeed-4ff1-848e-4b4702c06fa7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97" podUID="37f03155-aeed-4ff1-848e-4b4702c06fa7"
Jul 17 19:30:13.315130 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:13.315087    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97"]
Jul 17 19:30:13.337199 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:13.337161    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p"]
Jul 17 19:30:13.342071 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:13.342047    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p"
Jul 17 19:30:13.350182 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:13.350154    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p"]
Jul 17 19:30:13.457287 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:13.457267    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97"
Jul 17 19:30:13.486185 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:13.486161    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68070947-e654-4e7b-8a94-66d45cf99635-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p\" (UID: \"68070947-e654-4e7b-8a94-66d45cf99635\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p"
Jul 17 19:30:13.587697 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:13.587616    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37f03155-aeed-4ff1-848e-4b4702c06fa7-maas-api-service-ca\") pod \"37f03155-aeed-4ff1-848e-4b4702c06fa7\" (UID: \"37f03155-aeed-4ff1-848e-4b4702c06fa7\") "
Jul 17 19:30:13.587847 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:13.587768    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68070947-e654-4e7b-8a94-66d45cf99635-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p\" (UID: \"68070947-e654-4e7b-8a94-66d45cf99635\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p"
Jul 17 19:30:13.587980 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:13.587956    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f03155-aeed-4ff1-848e-4b4702c06fa7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "37f03155-aeed-4ff1-848e-4b4702c06fa7" (UID: "37f03155-aeed-4ff1-848e-4b4702c06fa7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:30:13.588338 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:13.588298    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68070947-e654-4e7b-8a94-66d45cf99635-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p\" (UID: \"68070947-e654-4e7b-8a94-66d45cf99635\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p"
Jul 17 19:30:13.656028 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:13.656002    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p"
Jul 17 19:30:13.688267 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:13.688243    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37f03155-aeed-4ff1-848e-4b4702c06fa7-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:30:13.792044 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:13.792017    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p"]
Jul 17 19:30:13.794146 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:30:13.794120    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68070947_e654_4e7b_8a94_66d45cf99635.slice/crio-23c35cac7b40e9255405004655b0feb4660237959affeae1ee15c79637f9a586 WatchSource:0}: Error finding container 23c35cac7b40e9255405004655b0feb4660237959affeae1ee15c79637f9a586: Status 404 returned error can't find the container with id 23c35cac7b40e9255405004655b0feb4660237959affeae1ee15c79637f9a586
Jul 17 19:30:14.300366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:14.300246    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p" event={"ID":"68070947-e654-4e7b-8a94-66d45cf99635","Type":"ContainerStarted","Data":"39833722f91099da1a1f3d23267a18dd3223855e2d771e91cc32ae6e4881bd65"}
Jul 17 19:30:14.300366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:14.300322    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p" event={"ID":"68070947-e654-4e7b-8a94-66d45cf99635","Type":"ContainerStarted","Data":"23c35cac7b40e9255405004655b0feb4660237959affeae1ee15c79637f9a586"}
Jul 17 19:30:14.301627 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:14.301603    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97"
Jul 17 19:30:14.301770 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:14.301662    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97" event={"ID":"37f03155-aeed-4ff1-848e-4b4702c06fa7","Type":"ContainerDied","Data":"b503b453b176934a1418736bb2b4d354351d872b6c665190ac9cfc76c69f5770"}
Jul 17 19:30:14.301770 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:14.301726    2559 scope.go:117] "RemoveContainer" containerID="f57caf6ab86a6bf9f000964a4b94a28b1e24ae0dd14ba0d16874039823f10cc0"
Jul 17 19:30:14.316972 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:14.316923    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p" podStartSLOduration=1.316907351 podStartE2EDuration="1.316907351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:30:14.313912067 +0000 UTC m=+3424.755420756" watchObservedRunningTime="2026-07-17 19:30:14.316907351 +0000 UTC m=+3424.758416040"
Jul 17 19:30:14.328360 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:14.328336    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97"]
Jul 17 19:30:14.333295 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:14.333267    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-v4f97"]
Jul 17 19:30:15.309398 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:15.309360    2559 generic.go:356] "Generic (PLEG): container finished" podID="68070947-e654-4e7b-8a94-66d45cf99635" containerID="39833722f91099da1a1f3d23267a18dd3223855e2d771e91cc32ae6e4881bd65" exitCode=6
Jul 17 19:30:15.309600 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:15.309454    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p" event={"ID":"68070947-e654-4e7b-8a94-66d45cf99635","Type":"ContainerDied","Data":"39833722f91099da1a1f3d23267a18dd3223855e2d771e91cc32ae6e4881bd65"}
Jul 17 19:30:15.309680 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:15.309638    2559 scope.go:117] "RemoveContainer" containerID="39833722f91099da1a1f3d23267a18dd3223855e2d771e91cc32ae6e4881bd65"
Jul 17 19:30:16.126424 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:16.126392    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37f03155-aeed-4ff1-848e-4b4702c06fa7" path="/var/lib/kubelet/pods/37f03155-aeed-4ff1-848e-4b4702c06fa7/volumes"
Jul 17 19:30:16.318248 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:16.318216    2559 generic.go:356] "Generic (PLEG): container finished" podID="68070947-e654-4e7b-8a94-66d45cf99635" containerID="e188b6797e0c6ec6685a09e21bc1e1803c655bc12e9ae2877a61c7a20f59c241" exitCode=6
Jul 17 19:30:16.318667 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:16.318285    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p" event={"ID":"68070947-e654-4e7b-8a94-66d45cf99635","Type":"ContainerDied","Data":"e188b6797e0c6ec6685a09e21bc1e1803c655bc12e9ae2877a61c7a20f59c241"}
Jul 17 19:30:16.318667 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:16.318381    2559 scope.go:117] "RemoveContainer" containerID="39833722f91099da1a1f3d23267a18dd3223855e2d771e91cc32ae6e4881bd65"
Jul 17 19:30:16.318667 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:16.318593    2559 scope.go:117] "RemoveContainer" containerID="e188b6797e0c6ec6685a09e21bc1e1803c655bc12e9ae2877a61c7a20f59c241"
Jul 17 19:30:16.318825 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:30:16.318769    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p_odh-ai-gateway-infra(68070947-e654-4e7b-8a94-66d45cf99635)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p" podUID="68070947-e654-4e7b-8a94-66d45cf99635"
Jul 17 19:30:17.326856 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:17.326833    2559 scope.go:117] "RemoveContainer" containerID="e188b6797e0c6ec6685a09e21bc1e1803c655bc12e9ae2877a61c7a20f59c241"
Jul 17 19:30:17.327251 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:30:17.326993    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p_odh-ai-gateway-infra(68070947-e654-4e7b-8a94-66d45cf99635)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p" podUID="68070947-e654-4e7b-8a94-66d45cf99635"
Jul 17 19:30:21.119730 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:21.119703    2559 scope.go:117] "RemoveContainer" containerID="ffceb2dffe99eba7d99fa409d78470a0a6cf097bd2ec99121599b5be23689161"
Jul 17 19:30:21.351084 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:21.351058    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265" event={"ID":"faa27fa0-d5e0-485c-8406-81d04e99044a","Type":"ContainerStarted","Data":"09b3e7fe4d3ad0bf912ec5ff45b115b8e511d600c763f59b10d29bcb44c4e8ec"}
Jul 17 19:30:21.367657 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:21.367614    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265" podStartSLOduration=17.367599803 podStartE2EDuration="17.367599803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:30:21.364594053 +0000 UTC m=+3431.806102754" watchObservedRunningTime="2026-07-17 19:30:21.367599803 +0000 UTC m=+3431.809108493"
Jul 17 19:30:22.358827 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:22.358792    2559 generic.go:356] "Generic (PLEG): container finished" podID="faa27fa0-d5e0-485c-8406-81d04e99044a" containerID="09b3e7fe4d3ad0bf912ec5ff45b115b8e511d600c763f59b10d29bcb44c4e8ec" exitCode=6
Jul 17 19:30:22.359275 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:22.358872    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265" event={"ID":"faa27fa0-d5e0-485c-8406-81d04e99044a","Type":"ContainerDied","Data":"09b3e7fe4d3ad0bf912ec5ff45b115b8e511d600c763f59b10d29bcb44c4e8ec"}
Jul 17 19:30:22.359275 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:22.358947    2559 scope.go:117] "RemoveContainer" containerID="ffceb2dffe99eba7d99fa409d78470a0a6cf097bd2ec99121599b5be23689161"
Jul 17 19:30:22.359275 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:22.359177    2559 scope.go:117] "RemoveContainer" containerID="09b3e7fe4d3ad0bf912ec5ff45b115b8e511d600c763f59b10d29bcb44c4e8ec"
Jul 17 19:30:22.359487 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:30:22.359464    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265_odh-ai-gateway-infra(faa27fa0-d5e0-485c-8406-81d04e99044a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265" podUID="faa27fa0-d5e0-485c-8406-81d04e99044a"
Jul 17 19:30:22.382818 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:22.382787    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265"]
Jul 17 19:30:23.497564 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:23.497539    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265"
Jul 17 19:30:23.579441 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:23.579407    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa27fa0-d5e0-485c-8406-81d04e99044a-maas-api-service-ca\") pod \"faa27fa0-d5e0-485c-8406-81d04e99044a\" (UID: \"faa27fa0-d5e0-485c-8406-81d04e99044a\") "
Jul 17 19:30:23.579722 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:23.579702    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa27fa0-d5e0-485c-8406-81d04e99044a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "faa27fa0-d5e0-485c-8406-81d04e99044a" (UID: "faa27fa0-d5e0-485c-8406-81d04e99044a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:30:23.680875 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:23.680794    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa27fa0-d5e0-485c-8406-81d04e99044a-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:30:24.373039 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.373009    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265"
Jul 17 19:30:24.373198 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.373034    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265" event={"ID":"faa27fa0-d5e0-485c-8406-81d04e99044a","Type":"ContainerDied","Data":"ca318e82682acd2ce81dc628555b591b67f4cc7789244b4fb77366cdccbd35f2"}
Jul 17 19:30:24.373198 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.373098    2559 scope.go:117] "RemoveContainer" containerID="09b3e7fe4d3ad0bf912ec5ff45b115b8e511d600c763f59b10d29bcb44c4e8ec"
Jul 17 19:30:24.389987 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.389903    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265"]
Jul 17 19:30:24.392850 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.392827    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-5l265"]
Jul 17 19:30:24.709236 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.709155    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"]
Jul 17 19:30:24.709789 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.709768    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faa27fa0-d5e0-485c-8406-81d04e99044a" containerName="revoke-keys"
Jul 17 19:30:24.709840 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.709795    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="faa27fa0-d5e0-485c-8406-81d04e99044a" containerName="revoke-keys"
Jul 17 19:30:24.709840 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.709816    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f03155-aeed-4ff1-848e-4b4702c06fa7" containerName="revoke-keys"
Jul 17 19:30:24.709840 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.709825    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f03155-aeed-4ff1-848e-4b4702c06fa7" containerName="revoke-keys"
Jul 17 19:30:24.709929 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.709843    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f03155-aeed-4ff1-848e-4b4702c06fa7" containerName="revoke-keys"
Jul 17 19:30:24.709929 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.709852    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f03155-aeed-4ff1-848e-4b4702c06fa7" containerName="revoke-keys"
Jul 17 19:30:24.709929 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.709871    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faa27fa0-d5e0-485c-8406-81d04e99044a" containerName="revoke-keys"
Jul 17 19:30:24.709929 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.709881    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="faa27fa0-d5e0-485c-8406-81d04e99044a" containerName="revoke-keys"
Jul 17 19:30:24.710047 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.709979    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="faa27fa0-d5e0-485c-8406-81d04e99044a" containerName="revoke-keys"
Jul 17 19:30:24.710047 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.709993    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f03155-aeed-4ff1-848e-4b4702c06fa7" containerName="revoke-keys"
Jul 17 19:30:24.710047 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.710003    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="faa27fa0-d5e0-485c-8406-81d04e99044a" containerName="revoke-keys"
Jul 17 19:30:24.710047 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.710015    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f03155-aeed-4ff1-848e-4b4702c06fa7" containerName="revoke-keys"
Jul 17 19:30:24.710047 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.710024    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f03155-aeed-4ff1-848e-4b4702c06fa7" containerName="revoke-keys"
Jul 17 19:30:24.714839 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.714817    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:24.717276 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.717250    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2109fd\"/\"sub-test-model-2109fd-kserve-self-signed-certs\""
Jul 17 19:30:24.718234 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.718215    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2109fd\"/\"openshift-service-ca.crt\""
Jul 17 19:30:24.718366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.718216    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2109fd\"/\"default-dockercfg-zf2rg\""
Jul 17 19:30:24.718366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.718261    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2109fd\"/\"kube-root-ca.crt\""
Jul 17 19:30:24.719016 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.718991    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"]
Jul 17 19:30:24.891034 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.891003    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-home\") pod \"sub-test-model-2109fd-kserve-65b78b775-pbrn5\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") " pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:24.891210 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.891058    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61afdfb6-a6fa-4461-8a2e-1494935a554d-tls-certs\") pod \"sub-test-model-2109fd-kserve-65b78b775-pbrn5\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") " pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:24.891210 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.891092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-model-cache\") pod \"sub-test-model-2109fd-kserve-65b78b775-pbrn5\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") " pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:24.891210 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.891115    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-tmp-dir\") pod \"sub-test-model-2109fd-kserve-65b78b775-pbrn5\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") " pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:24.891210 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.891168    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slbjc\" (UniqueName: \"kubernetes.io/projected/61afdfb6-a6fa-4461-8a2e-1494935a554d-kube-api-access-slbjc\") pod \"sub-test-model-2109fd-kserve-65b78b775-pbrn5\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") " pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:24.891395 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.891216    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-dshm\") pod \"sub-test-model-2109fd-kserve-65b78b775-pbrn5\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") " pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:24.991935 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.991869    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-home\") pod \"sub-test-model-2109fd-kserve-65b78b775-pbrn5\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") " pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:24.991935 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.991916    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61afdfb6-a6fa-4461-8a2e-1494935a554d-tls-certs\") pod \"sub-test-model-2109fd-kserve-65b78b775-pbrn5\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") " pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:24.991935 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.991936    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-model-cache\") pod \"sub-test-model-2109fd-kserve-65b78b775-pbrn5\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") " pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:24.992196 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.992062    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-tmp-dir\") pod \"sub-test-model-2109fd-kserve-65b78b775-pbrn5\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") " pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:24.992196 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.992119    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slbjc\" (UniqueName: \"kubernetes.io/projected/61afdfb6-a6fa-4461-8a2e-1494935a554d-kube-api-access-slbjc\") pod \"sub-test-model-2109fd-kserve-65b78b775-pbrn5\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") " pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:24.992196 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.992175    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-dshm\") pod \"sub-test-model-2109fd-kserve-65b78b775-pbrn5\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") " pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:24.992358 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.992260    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-home\") pod \"sub-test-model-2109fd-kserve-65b78b775-pbrn5\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") " pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:24.992358 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.992331    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-model-cache\") pod \"sub-test-model-2109fd-kserve-65b78b775-pbrn5\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") " pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:24.992435 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.992362    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-tmp-dir\") pod \"sub-test-model-2109fd-kserve-65b78b775-pbrn5\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") " pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:24.994394 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.994375    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-dshm\") pod \"sub-test-model-2109fd-kserve-65b78b775-pbrn5\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") " pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:24.994593 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:24.994577    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61afdfb6-a6fa-4461-8a2e-1494935a554d-tls-certs\") pod \"sub-test-model-2109fd-kserve-65b78b775-pbrn5\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") " pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:25.001178 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:25.001158    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slbjc\" (UniqueName: \"kubernetes.io/projected/61afdfb6-a6fa-4461-8a2e-1494935a554d-kube-api-access-slbjc\") pod \"sub-test-model-2109fd-kserve-65b78b775-pbrn5\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") " pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:25.026199 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:25.026178    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:25.163110 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:25.163079    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"]
Jul 17 19:30:25.163794 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:30:25.163767    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61afdfb6_a6fa_4461_8a2e_1494935a554d.slice/crio-6c9ab78d0a648289f2d190d9ccb398c0d7d66915ed265d91618f6bef6e49999a WatchSource:0}: Error finding container 6c9ab78d0a648289f2d190d9ccb398c0d7d66915ed265d91618f6bef6e49999a: Status 404 returned error can't find the container with id 6c9ab78d0a648289f2d190d9ccb398c0d7d66915ed265d91618f6bef6e49999a
Jul 17 19:30:25.381056 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:25.381018    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5" event={"ID":"61afdfb6-a6fa-4461-8a2e-1494935a554d","Type":"ContainerStarted","Data":"c074fc165ddf6011a2b0566477886e2ee9bee5e00f69a50768c667cb815682b3"}
Jul 17 19:30:25.381056 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:25.381060    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5" event={"ID":"61afdfb6-a6fa-4461-8a2e-1494935a554d","Type":"ContainerStarted","Data":"6c9ab78d0a648289f2d190d9ccb398c0d7d66915ed265d91618f6bef6e49999a"}
Jul 17 19:30:25.398963 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:25.398914    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5" podStartSLOduration=1.3988997539999999 podStartE2EDuration="1.398899754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:30:25.397610728 +0000 UTC m=+3435.839119419" watchObservedRunningTime="2026-07-17 19:30:25.398899754 +0000 UTC m=+3435.840408491"
Jul 17 19:30:26.126363 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:26.126333    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faa27fa0-d5e0-485c-8406-81d04e99044a" path="/var/lib/kubelet/pods/faa27fa0-d5e0-485c-8406-81d04e99044a/volumes"
Jul 17 19:30:31.119831 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:31.119803    2559 scope.go:117] "RemoveContainer" containerID="e188b6797e0c6ec6685a09e21bc1e1803c655bc12e9ae2877a61c7a20f59c241"
Jul 17 19:30:31.421486 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:31.421417    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p" event={"ID":"68070947-e654-4e7b-8a94-66d45cf99635","Type":"ContainerStarted","Data":"2445793220dc5bead8f43eab4962966407eb720df6d6860399338fdaaa0c8e67"}
Jul 17 19:30:32.428982 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:32.428946    2559 generic.go:356] "Generic (PLEG): container finished" podID="68070947-e654-4e7b-8a94-66d45cf99635" containerID="2445793220dc5bead8f43eab4962966407eb720df6d6860399338fdaaa0c8e67" exitCode=6
Jul 17 19:30:32.429437 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:32.429020    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p" event={"ID":"68070947-e654-4e7b-8a94-66d45cf99635","Type":"ContainerDied","Data":"2445793220dc5bead8f43eab4962966407eb720df6d6860399338fdaaa0c8e67"}
Jul 17 19:30:32.429437 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:32.429081    2559 scope.go:117] "RemoveContainer" containerID="e188b6797e0c6ec6685a09e21bc1e1803c655bc12e9ae2877a61c7a20f59c241"
Jul 17 19:30:32.429437 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:32.429391    2559 scope.go:117] "RemoveContainer" containerID="2445793220dc5bead8f43eab4962966407eb720df6d6860399338fdaaa0c8e67"
Jul 17 19:30:32.429625 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:30:32.429604    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p_odh-ai-gateway-infra(68070947-e654-4e7b-8a94-66d45cf99635)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p" podUID="68070947-e654-4e7b-8a94-66d45cf99635"
Jul 17 19:30:32.455231 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:32.455203    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p"]
Jul 17 19:30:33.571993 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:33.571964    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p"
Jul 17 19:30:33.668427 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:33.668394    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68070947-e654-4e7b-8a94-66d45cf99635-maas-api-service-ca\") pod \"68070947-e654-4e7b-8a94-66d45cf99635\" (UID: \"68070947-e654-4e7b-8a94-66d45cf99635\") "
Jul 17 19:30:33.668729 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:33.668706    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68070947-e654-4e7b-8a94-66d45cf99635-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "68070947-e654-4e7b-8a94-66d45cf99635" (UID: "68070947-e654-4e7b-8a94-66d45cf99635"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:30:33.769909 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:33.769827    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68070947-e654-4e7b-8a94-66d45cf99635-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:30:34.444287 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:34.444202    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p"
Jul 17 19:30:34.444287 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:34.444245    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p" event={"ID":"68070947-e654-4e7b-8a94-66d45cf99635","Type":"ContainerDied","Data":"23c35cac7b40e9255405004655b0feb4660237959affeae1ee15c79637f9a586"}
Jul 17 19:30:34.444506 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:34.444340    2559 scope.go:117] "RemoveContainer" containerID="2445793220dc5bead8f43eab4962966407eb720df6d6860399338fdaaa0c8e67"
Jul 17 19:30:34.465088 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:34.465035    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p"]
Jul 17 19:30:34.468125 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:34.468105    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-76n5p"]
Jul 17 19:30:35.026724 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:35.026688    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:35.026724 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:35.026730    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:35.039728 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:35.039704    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:35.463409 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:35.463380    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:30:36.125886 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:36.125846    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68070947-e654-4e7b-8a94-66d45cf99635" path="/var/lib/kubelet/pods/68070947-e654-4e7b-8a94-66d45cf99635/volumes"
Jul 17 19:30:41.614825 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.614788    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r"]
Jul 17 19:30:41.617381 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.615410    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f03155-aeed-4ff1-848e-4b4702c06fa7" containerName="revoke-keys"
Jul 17 19:30:41.617381 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.615430    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f03155-aeed-4ff1-848e-4b4702c06fa7" containerName="revoke-keys"
Jul 17 19:30:41.617381 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.615461    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68070947-e654-4e7b-8a94-66d45cf99635" containerName="revoke-keys"
Jul 17 19:30:41.617381 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.615470    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="68070947-e654-4e7b-8a94-66d45cf99635" containerName="revoke-keys"
Jul 17 19:30:41.617381 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.615486    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faa27fa0-d5e0-485c-8406-81d04e99044a" containerName="revoke-keys"
Jul 17 19:30:41.617381 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.615494    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="faa27fa0-d5e0-485c-8406-81d04e99044a" containerName="revoke-keys"
Jul 17 19:30:41.617381 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.615509    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68070947-e654-4e7b-8a94-66d45cf99635" containerName="revoke-keys"
Jul 17 19:30:41.617381 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.615517    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="68070947-e654-4e7b-8a94-66d45cf99635" containerName="revoke-keys"
Jul 17 19:30:41.617381 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.615528    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68070947-e654-4e7b-8a94-66d45cf99635" containerName="revoke-keys"
Jul 17 19:30:41.617381 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.615534    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="68070947-e654-4e7b-8a94-66d45cf99635" containerName="revoke-keys"
Jul 17 19:30:41.617381 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.615621    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="68070947-e654-4e7b-8a94-66d45cf99635" containerName="revoke-keys"
Jul 17 19:30:41.617381 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.615630    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="faa27fa0-d5e0-485c-8406-81d04e99044a" containerName="revoke-keys"
Jul 17 19:30:41.617381 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.615637    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="68070947-e654-4e7b-8a94-66d45cf99635" containerName="revoke-keys"
Jul 17 19:30:41.617381 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.615642    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="68070947-e654-4e7b-8a94-66d45cf99635" containerName="revoke-keys"
Jul 17 19:30:41.618911 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.618891    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r"]
Jul 17 19:30:41.619045 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.618973    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r"
Jul 17 19:30:41.621914 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.621893    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:30:41.734892 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.734868    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7eaedd4-df54-44fa-9410-815a63230e10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r\" (UID: \"b7eaedd4-df54-44fa-9410-815a63230e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r"
Jul 17 19:30:41.836074 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.836047    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7eaedd4-df54-44fa-9410-815a63230e10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r\" (UID: \"b7eaedd4-df54-44fa-9410-815a63230e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r"
Jul 17 19:30:41.836636 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.836615    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7eaedd4-df54-44fa-9410-815a63230e10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r\" (UID: \"b7eaedd4-df54-44fa-9410-815a63230e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r"
Jul 17 19:30:41.929418 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:41.929350    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r"
Jul 17 19:30:42.065399 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:30:42.065369    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7eaedd4_df54_44fa_9410_815a63230e10.slice/crio-96fcbf7696516259e0c0e9c654d1c2fbb9927396785b9a64330543d7e565b80d WatchSource:0}: Error finding container 96fcbf7696516259e0c0e9c654d1c2fbb9927396785b9a64330543d7e565b80d: Status 404 returned error can't find the container with id 96fcbf7696516259e0c0e9c654d1c2fbb9927396785b9a64330543d7e565b80d
Jul 17 19:30:42.065515 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:42.065454    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r"]
Jul 17 19:30:42.493393 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:42.493353    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r" event={"ID":"b7eaedd4-df54-44fa-9410-815a63230e10","Type":"ContainerStarted","Data":"4618f0424cecbb938a343e7544135f5bf59cbeb415eedede87348961f89a02e8"}
Jul 17 19:30:42.493770 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:42.493418    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r" event={"ID":"b7eaedd4-df54-44fa-9410-815a63230e10","Type":"ContainerStarted","Data":"96fcbf7696516259e0c0e9c654d1c2fbb9927396785b9a64330543d7e565b80d"}
Jul 17 19:30:42.509837 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:42.509755    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r" podStartSLOduration=1.5097411570000001 podStartE2EDuration="1.509741157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:30:42.508100278 +0000 UTC m=+3452.949608968" watchObservedRunningTime="2026-07-17 19:30:42.509741157 +0000 UTC m=+3452.951249847"
Jul 17 19:30:43.500952 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:43.500921    2559 generic.go:356] "Generic (PLEG): container finished" podID="b7eaedd4-df54-44fa-9410-815a63230e10" containerID="4618f0424cecbb938a343e7544135f5bf59cbeb415eedede87348961f89a02e8" exitCode=6
Jul 17 19:30:43.501425 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:43.501022    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r" event={"ID":"b7eaedd4-df54-44fa-9410-815a63230e10","Type":"ContainerDied","Data":"4618f0424cecbb938a343e7544135f5bf59cbeb415eedede87348961f89a02e8"}
Jul 17 19:30:43.501425 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:43.501189    2559 scope.go:117] "RemoveContainer" containerID="4618f0424cecbb938a343e7544135f5bf59cbeb415eedede87348961f89a02e8"
Jul 17 19:30:44.509814 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:44.509720    2559 generic.go:356] "Generic (PLEG): container finished" podID="b7eaedd4-df54-44fa-9410-815a63230e10" containerID="0d68c362c6a27b8435cfb844df2f3a6cb0cd85fc5cc27f343ad6b6c82a590ca8" exitCode=6
Jul 17 19:30:44.510238 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:44.509808    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r" event={"ID":"b7eaedd4-df54-44fa-9410-815a63230e10","Type":"ContainerDied","Data":"0d68c362c6a27b8435cfb844df2f3a6cb0cd85fc5cc27f343ad6b6c82a590ca8"}
Jul 17 19:30:44.510238 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:44.509867    2559 scope.go:117] "RemoveContainer" containerID="4618f0424cecbb938a343e7544135f5bf59cbeb415eedede87348961f89a02e8"
Jul 17 19:30:44.510238 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:44.510094    2559 scope.go:117] "RemoveContainer" containerID="0d68c362c6a27b8435cfb844df2f3a6cb0cd85fc5cc27f343ad6b6c82a590ca8"
Jul 17 19:30:44.510396 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:30:44.510346    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r_odh-ai-gateway-infra(b7eaedd4-df54-44fa-9410-815a63230e10)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r" podUID="b7eaedd4-df54-44fa-9410-815a63230e10"
Jul 17 19:30:45.518670 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:45.518648    2559 scope.go:117] "RemoveContainer" containerID="0d68c362c6a27b8435cfb844df2f3a6cb0cd85fc5cc27f343ad6b6c82a590ca8"
Jul 17 19:30:45.519045 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:30:45.518826    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r_odh-ai-gateway-infra(b7eaedd4-df54-44fa-9410-815a63230e10)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r" podUID="b7eaedd4-df54-44fa-9410-815a63230e10"
Jul 17 19:30:46.292204 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:46.292175    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7"]
Jul 17 19:30:46.296751 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:46.296730    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7"
Jul 17 19:30:46.303335 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:46.303292    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7"]
Jul 17 19:30:46.371962 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:46.371929    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/776e39e9-1624-4974-8202-86caec2dbfbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7\" (UID: \"776e39e9-1624-4974-8202-86caec2dbfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7"
Jul 17 19:30:46.472729 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:46.472698    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/776e39e9-1624-4974-8202-86caec2dbfbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7\" (UID: \"776e39e9-1624-4974-8202-86caec2dbfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7"
Jul 17 19:30:46.473246 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:46.473229    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/776e39e9-1624-4974-8202-86caec2dbfbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7\" (UID: \"776e39e9-1624-4974-8202-86caec2dbfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7"
Jul 17 19:30:46.608801 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:46.608776    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7"
Jul 17 19:30:46.745583 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:46.745555    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7"]
Jul 17 19:30:46.747000 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:30:46.746978    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776e39e9_1624_4974_8202_86caec2dbfbd.slice/crio-bb51c8006b70e4e1f7c486b133be651c30d8a0d962cc7c8c83b7564de9560b58 WatchSource:0}: Error finding container bb51c8006b70e4e1f7c486b133be651c30d8a0d962cc7c8c83b7564de9560b58: Status 404 returned error can't find the container with id bb51c8006b70e4e1f7c486b133be651c30d8a0d962cc7c8c83b7564de9560b58
Jul 17 19:30:47.532053 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:47.532019    2559 generic.go:356] "Generic (PLEG): container finished" podID="776e39e9-1624-4974-8202-86caec2dbfbd" containerID="4128fd98c627d8d21d2e3559ef146e825bcc948eac4a1bed58d117609fad0c18" exitCode=6
Jul 17 19:30:47.532284 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:47.532058    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7" event={"ID":"776e39e9-1624-4974-8202-86caec2dbfbd","Type":"ContainerDied","Data":"4128fd98c627d8d21d2e3559ef146e825bcc948eac4a1bed58d117609fad0c18"}
Jul 17 19:30:47.532284 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:47.532095    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7" event={"ID":"776e39e9-1624-4974-8202-86caec2dbfbd","Type":"ContainerStarted","Data":"bb51c8006b70e4e1f7c486b133be651c30d8a0d962cc7c8c83b7564de9560b58"}
Jul 17 19:30:47.532441 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:47.532300    2559 scope.go:117] "RemoveContainer" containerID="4128fd98c627d8d21d2e3559ef146e825bcc948eac4a1bed58d117609fad0c18"
Jul 17 19:30:48.539742 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:48.539659    2559 generic.go:356] "Generic (PLEG): container finished" podID="776e39e9-1624-4974-8202-86caec2dbfbd" containerID="232f4ce4763d7d799b57a8823b5f9fd690dc013a3655cd0213be6df5267d8b31" exitCode=6
Jul 17 19:30:48.540158 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:48.539780    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7" event={"ID":"776e39e9-1624-4974-8202-86caec2dbfbd","Type":"ContainerDied","Data":"232f4ce4763d7d799b57a8823b5f9fd690dc013a3655cd0213be6df5267d8b31"}
Jul 17 19:30:48.540158 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:48.539833    2559 scope.go:117] "RemoveContainer" containerID="4128fd98c627d8d21d2e3559ef146e825bcc948eac4a1bed58d117609fad0c18"
Jul 17 19:30:48.540158 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:48.539887    2559 scope.go:117] "RemoveContainer" containerID="232f4ce4763d7d799b57a8823b5f9fd690dc013a3655cd0213be6df5267d8b31"
Jul 17 19:30:48.540286 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:30:48.540149    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7_odh-ai-gateway-infra(776e39e9-1624-4974-8202-86caec2dbfbd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7" podUID="776e39e9-1624-4974-8202-86caec2dbfbd"
Jul 17 19:30:49.547141 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:30:49.547115    2559 scope.go:117] "RemoveContainer" containerID="232f4ce4763d7d799b57a8823b5f9fd690dc013a3655cd0213be6df5267d8b31"
Jul 17 19:30:49.547516 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:30:49.547355    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7_odh-ai-gateway-infra(776e39e9-1624-4974-8202-86caec2dbfbd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7" podUID="776e39e9-1624-4974-8202-86caec2dbfbd"
Jul 17 19:31:00.123427 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:00.123394    2559 scope.go:117] "RemoveContainer" containerID="0d68c362c6a27b8435cfb844df2f3a6cb0cd85fc5cc27f343ad6b6c82a590ca8"
Jul 17 19:31:00.613740 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:00.613701    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r" event={"ID":"b7eaedd4-df54-44fa-9410-815a63230e10","Type":"ContainerStarted","Data":"8c2330ce4a421d854cb8a602d633094352509ac12ce893cc023b3d7921f01c60"}
Jul 17 19:31:01.621600 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:01.621565    2559 generic.go:356] "Generic (PLEG): container finished" podID="b7eaedd4-df54-44fa-9410-815a63230e10" containerID="8c2330ce4a421d854cb8a602d633094352509ac12ce893cc023b3d7921f01c60" exitCode=6
Jul 17 19:31:01.622096 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:01.621643    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r" event={"ID":"b7eaedd4-df54-44fa-9410-815a63230e10","Type":"ContainerDied","Data":"8c2330ce4a421d854cb8a602d633094352509ac12ce893cc023b3d7921f01c60"}
Jul 17 19:31:01.622096 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:01.621721    2559 scope.go:117] "RemoveContainer" containerID="0d68c362c6a27b8435cfb844df2f3a6cb0cd85fc5cc27f343ad6b6c82a590ca8"
Jul 17 19:31:01.622096 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:01.622045    2559 scope.go:117] "RemoveContainer" containerID="8c2330ce4a421d854cb8a602d633094352509ac12ce893cc023b3d7921f01c60"
Jul 17 19:31:01.622268 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:31:01.622244    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r_odh-ai-gateway-infra(b7eaedd4-df54-44fa-9410-815a63230e10)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r" podUID="b7eaedd4-df54-44fa-9410-815a63230e10"
Jul 17 19:31:01.645659 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:01.645636    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r"]
Jul 17 19:31:02.768664 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:02.768641    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r"
Jul 17 19:31:02.929876 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:02.929779    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7eaedd4-df54-44fa-9410-815a63230e10-maas-api-service-ca\") pod \"b7eaedd4-df54-44fa-9410-815a63230e10\" (UID: \"b7eaedd4-df54-44fa-9410-815a63230e10\") "
Jul 17 19:31:02.930192 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:02.930170    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7eaedd4-df54-44fa-9410-815a63230e10-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b7eaedd4-df54-44fa-9410-815a63230e10" (UID: "b7eaedd4-df54-44fa-9410-815a63230e10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:31:03.031059 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:03.031014    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7eaedd4-df54-44fa-9410-815a63230e10-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:31:03.126239 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:03.126206    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"]
Jul 17 19:31:03.643068 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:03.643034    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r"
Jul 17 19:31:03.643278 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:03.643064    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r" event={"ID":"b7eaedd4-df54-44fa-9410-815a63230e10","Type":"ContainerDied","Data":"96fcbf7696516259e0c0e9c654d1c2fbb9927396785b9a64330543d7e565b80d"}
Jul 17 19:31:03.643278 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:03.643164    2559 scope.go:117] "RemoveContainer" containerID="8c2330ce4a421d854cb8a602d633094352509ac12ce893cc023b3d7921f01c60"
Jul 17 19:31:03.666856 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:03.666829    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r"]
Jul 17 19:31:03.672484 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:03.672461    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-cbv9r"]
Jul 17 19:31:04.120175 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:04.120143    2559 scope.go:117] "RemoveContainer" containerID="232f4ce4763d7d799b57a8823b5f9fd690dc013a3655cd0213be6df5267d8b31"
Jul 17 19:31:04.125880 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:04.125857    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7eaedd4-df54-44fa-9410-815a63230e10" path="/var/lib/kubelet/pods/b7eaedd4-df54-44fa-9410-815a63230e10/volumes"
Jul 17 19:31:04.651023 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:04.650990    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7" event={"ID":"776e39e9-1624-4974-8202-86caec2dbfbd","Type":"ContainerStarted","Data":"c02e7fd397e0a28e4739fe397595e9d04d39048b2dcbddcf4e06735faf60a6fa"}
Jul 17 19:31:04.669144 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:04.669094    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7" podStartSLOduration=18.669079714 podStartE2EDuration="18.669079714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:31:04.666247224 +0000 UTC m=+3475.107755914" watchObservedRunningTime="2026-07-17 19:31:04.669079714 +0000 UTC m=+3475.110588402"
Jul 17 19:31:05.660070 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:05.660036    2559 generic.go:356] "Generic (PLEG): container finished" podID="776e39e9-1624-4974-8202-86caec2dbfbd" containerID="c02e7fd397e0a28e4739fe397595e9d04d39048b2dcbddcf4e06735faf60a6fa" exitCode=6
Jul 17 19:31:05.660530 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:05.660100    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7" event={"ID":"776e39e9-1624-4974-8202-86caec2dbfbd","Type":"ContainerDied","Data":"c02e7fd397e0a28e4739fe397595e9d04d39048b2dcbddcf4e06735faf60a6fa"}
Jul 17 19:31:05.660530 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:05.660157    2559 scope.go:117] "RemoveContainer" containerID="232f4ce4763d7d799b57a8823b5f9fd690dc013a3655cd0213be6df5267d8b31"
Jul 17 19:31:05.660530 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:05.660508    2559 scope.go:117] "RemoveContainer" containerID="c02e7fd397e0a28e4739fe397595e9d04d39048b2dcbddcf4e06735faf60a6fa"
Jul 17 19:31:05.660740 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:31:05.660720    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7_odh-ai-gateway-infra(776e39e9-1624-4974-8202-86caec2dbfbd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7" podUID="776e39e9-1624-4974-8202-86caec2dbfbd"
Jul 17 19:31:05.676798 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:05.676775    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7"]
Jul 17 19:31:06.801019 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:06.800996    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7"
Jul 17 19:31:06.969446 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:06.969366    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/776e39e9-1624-4974-8202-86caec2dbfbd-maas-api-service-ca\") pod \"776e39e9-1624-4974-8202-86caec2dbfbd\" (UID: \"776e39e9-1624-4974-8202-86caec2dbfbd\") "
Jul 17 19:31:06.969712 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:06.969690    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/776e39e9-1624-4974-8202-86caec2dbfbd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "776e39e9-1624-4974-8202-86caec2dbfbd" (UID: "776e39e9-1624-4974-8202-86caec2dbfbd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:31:07.070598 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:07.070567    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/776e39e9-1624-4974-8202-86caec2dbfbd-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:31:07.675456 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:07.675431    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7"
Jul 17 19:31:07.675456 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:07.675434    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7" event={"ID":"776e39e9-1624-4974-8202-86caec2dbfbd","Type":"ContainerDied","Data":"bb51c8006b70e4e1f7c486b133be651c30d8a0d962cc7c8c83b7564de9560b58"}
Jul 17 19:31:07.675720 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:07.675491    2559 scope.go:117] "RemoveContainer" containerID="c02e7fd397e0a28e4739fe397595e9d04d39048b2dcbddcf4e06735faf60a6fa"
Jul 17 19:31:07.699225 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:07.699199    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7"]
Jul 17 19:31:07.707059 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:07.707023    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2jht7"]
Jul 17 19:31:08.129121 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:08.129086    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="776e39e9-1624-4974-8202-86caec2dbfbd" path="/var/lib/kubelet/pods/776e39e9-1624-4974-8202-86caec2dbfbd/volumes"
Jul 17 19:31:13.384676 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.384643    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5"]
Jul 17 19:31:13.385249 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.385229    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776e39e9-1624-4974-8202-86caec2dbfbd" containerName="revoke-keys"
Jul 17 19:31:13.385300 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.385255    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="776e39e9-1624-4974-8202-86caec2dbfbd" containerName="revoke-keys"
Jul 17 19:31:13.385300 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.385274    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7eaedd4-df54-44fa-9410-815a63230e10" containerName="revoke-keys"
Jul 17 19:31:13.385300 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.385283    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7eaedd4-df54-44fa-9410-815a63230e10" containerName="revoke-keys"
Jul 17 19:31:13.385300 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.385320    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776e39e9-1624-4974-8202-86caec2dbfbd" containerName="revoke-keys"
Jul 17 19:31:13.385483 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.385330    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="776e39e9-1624-4974-8202-86caec2dbfbd" containerName="revoke-keys"
Jul 17 19:31:13.385483 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.385344    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7eaedd4-df54-44fa-9410-815a63230e10" containerName="revoke-keys"
Jul 17 19:31:13.385483 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.385352    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7eaedd4-df54-44fa-9410-815a63230e10" containerName="revoke-keys"
Jul 17 19:31:13.385483 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.385476    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="776e39e9-1624-4974-8202-86caec2dbfbd" containerName="revoke-keys"
Jul 17 19:31:13.385602 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.385488    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7eaedd4-df54-44fa-9410-815a63230e10" containerName="revoke-keys"
Jul 17 19:31:13.385602 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.385498    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="776e39e9-1624-4974-8202-86caec2dbfbd" containerName="revoke-keys"
Jul 17 19:31:13.385602 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.385506    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7eaedd4-df54-44fa-9410-815a63230e10" containerName="revoke-keys"
Jul 17 19:31:13.385602 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.385516    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="776e39e9-1624-4974-8202-86caec2dbfbd" containerName="revoke-keys"
Jul 17 19:31:13.390698 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.390681    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5"
Jul 17 19:31:13.393240 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.393218    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:31:13.395666 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.395641    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5"]
Jul 17 19:31:13.530659 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.530610    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c542e7a9-8812-433b-a6e1-d72d204abdfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5\" (UID: \"c542e7a9-8812-433b-a6e1-d72d204abdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5"
Jul 17 19:31:13.631513 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.631476    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c542e7a9-8812-433b-a6e1-d72d204abdfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5\" (UID: \"c542e7a9-8812-433b-a6e1-d72d204abdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5"
Jul 17 19:31:13.632046 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.632026    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c542e7a9-8812-433b-a6e1-d72d204abdfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5\" (UID: \"c542e7a9-8812-433b-a6e1-d72d204abdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5"
Jul 17 19:31:13.702836 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.702769    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5"
Jul 17 19:31:13.837878 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:13.837800    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5"]
Jul 17 19:31:13.840098 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:31:13.840066    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc542e7a9_8812_433b_a6e1_d72d204abdfe.slice/crio-6b5c8550f013b8cc144692bf766671507ddb8dce9acf5ca5f25bff9d33d71fe9 WatchSource:0}: Error finding container 6b5c8550f013b8cc144692bf766671507ddb8dce9acf5ca5f25bff9d33d71fe9: Status 404 returned error can't find the container with id 6b5c8550f013b8cc144692bf766671507ddb8dce9acf5ca5f25bff9d33d71fe9
Jul 17 19:31:14.719240 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:14.719145    2559 generic.go:356] "Generic (PLEG): container finished" podID="c542e7a9-8812-433b-a6e1-d72d204abdfe" containerID="be27b70e0bb9b91314b3c8f066ff712e2ad6dc0166623eaa06d66abd1e2129c1" exitCode=6
Jul 17 19:31:14.719705 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:14.719231    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5" event={"ID":"c542e7a9-8812-433b-a6e1-d72d204abdfe","Type":"ContainerDied","Data":"be27b70e0bb9b91314b3c8f066ff712e2ad6dc0166623eaa06d66abd1e2129c1"}
Jul 17 19:31:14.719705 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:14.719343    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5" event={"ID":"c542e7a9-8812-433b-a6e1-d72d204abdfe","Type":"ContainerStarted","Data":"6b5c8550f013b8cc144692bf766671507ddb8dce9acf5ca5f25bff9d33d71fe9"}
Jul 17 19:31:14.719705 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:14.719468    2559 scope.go:117] "RemoveContainer" containerID="be27b70e0bb9b91314b3c8f066ff712e2ad6dc0166623eaa06d66abd1e2129c1"
Jul 17 19:31:15.727339 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:15.727219    2559 generic.go:356] "Generic (PLEG): container finished" podID="c542e7a9-8812-433b-a6e1-d72d204abdfe" containerID="74ce38170f4d286f90bd21210d417eac806d6ed0e4117116545b79f3050f4644" exitCode=6
Jul 17 19:31:15.727767 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:15.727329    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5" event={"ID":"c542e7a9-8812-433b-a6e1-d72d204abdfe","Type":"ContainerDied","Data":"74ce38170f4d286f90bd21210d417eac806d6ed0e4117116545b79f3050f4644"}
Jul 17 19:31:15.727767 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:15.727392    2559 scope.go:117] "RemoveContainer" containerID="be27b70e0bb9b91314b3c8f066ff712e2ad6dc0166623eaa06d66abd1e2129c1"
Jul 17 19:31:15.727767 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:15.727624    2559 scope.go:117] "RemoveContainer" containerID="74ce38170f4d286f90bd21210d417eac806d6ed0e4117116545b79f3050f4644"
Jul 17 19:31:15.727919 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:31:15.727851    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5_odh-ai-gateway-infra(c542e7a9-8812-433b-a6e1-d72d204abdfe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5" podUID="c542e7a9-8812-433b-a6e1-d72d204abdfe"
Jul 17 19:31:16.734484 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:16.734458    2559 scope.go:117] "RemoveContainer" containerID="74ce38170f4d286f90bd21210d417eac806d6ed0e4117116545b79f3050f4644"
Jul 17 19:31:16.734874 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:31:16.734664    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5_odh-ai-gateway-infra(c542e7a9-8812-433b-a6e1-d72d204abdfe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5" podUID="c542e7a9-8812-433b-a6e1-d72d204abdfe"
Jul 17 19:31:18.146959 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.146919    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5" podUID="61afdfb6-a6fa-4461-8a2e-1494935a554d" containerName="main" containerID="cri-o://c074fc165ddf6011a2b0566477886e2ee9bee5e00f69a50768c667cb815682b3" gracePeriod=45
Jul 17 19:31:18.398883 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.398829    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:31:18.478179 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.478149    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-tmp-dir\") pod \"61afdfb6-a6fa-4461-8a2e-1494935a554d\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") "
Jul 17 19:31:18.478367 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.478190    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slbjc\" (UniqueName: \"kubernetes.io/projected/61afdfb6-a6fa-4461-8a2e-1494935a554d-kube-api-access-slbjc\") pod \"61afdfb6-a6fa-4461-8a2e-1494935a554d\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") "
Jul 17 19:31:18.478367 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.478225    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61afdfb6-a6fa-4461-8a2e-1494935a554d-tls-certs\") pod \"61afdfb6-a6fa-4461-8a2e-1494935a554d\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") "
Jul 17 19:31:18.478367 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.478278    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-home\") pod \"61afdfb6-a6fa-4461-8a2e-1494935a554d\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") "
Jul 17 19:31:18.478367 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.478330    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-dshm\") pod \"61afdfb6-a6fa-4461-8a2e-1494935a554d\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") "
Jul 17 19:31:18.478367 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.478366    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-model-cache\") pod \"61afdfb6-a6fa-4461-8a2e-1494935a554d\" (UID: \"61afdfb6-a6fa-4461-8a2e-1494935a554d\") "
Jul 17 19:31:18.478650 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.478471    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61afdfb6-a6fa-4461-8a2e-1494935a554d" (UID: "61afdfb6-a6fa-4461-8a2e-1494935a554d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:31:18.478650 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.478576    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-home" (OuterVolumeSpecName: "home") pod "61afdfb6-a6fa-4461-8a2e-1494935a554d" (UID: "61afdfb6-a6fa-4461-8a2e-1494935a554d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:31:18.478650 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.478642    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-model-cache" (OuterVolumeSpecName: "model-cache") pod "61afdfb6-a6fa-4461-8a2e-1494935a554d" (UID: "61afdfb6-a6fa-4461-8a2e-1494935a554d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:31:18.478804 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.478657    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-home\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:31:18.478804 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.478673    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-tmp-dir\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:31:18.480472 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.480440    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61afdfb6-a6fa-4461-8a2e-1494935a554d-kube-api-access-slbjc" (OuterVolumeSpecName: "kube-api-access-slbjc") pod "61afdfb6-a6fa-4461-8a2e-1494935a554d" (UID: "61afdfb6-a6fa-4461-8a2e-1494935a554d"). InnerVolumeSpecName "kube-api-access-slbjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:31:18.480472 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.480448    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61afdfb6-a6fa-4461-8a2e-1494935a554d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61afdfb6-a6fa-4461-8a2e-1494935a554d" (UID: "61afdfb6-a6fa-4461-8a2e-1494935a554d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:31:18.480605 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.480477    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-dshm" (OuterVolumeSpecName: "dshm") pod "61afdfb6-a6fa-4461-8a2e-1494935a554d" (UID: "61afdfb6-a6fa-4461-8a2e-1494935a554d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:31:18.579666 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.579639    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-dshm\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:31:18.579666 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.579664    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61afdfb6-a6fa-4461-8a2e-1494935a554d-model-cache\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:31:18.579836 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.579676    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slbjc\" (UniqueName: \"kubernetes.io/projected/61afdfb6-a6fa-4461-8a2e-1494935a554d-kube-api-access-slbjc\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:31:18.579836 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.579685    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61afdfb6-a6fa-4461-8a2e-1494935a554d-tls-certs\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:31:18.607481 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.607456    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"]
Jul 17 19:31:18.607844 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.607832    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7eaedd4-df54-44fa-9410-815a63230e10" containerName="revoke-keys"
Jul 17 19:31:18.607889 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.607846    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7eaedd4-df54-44fa-9410-815a63230e10" containerName="revoke-keys"
Jul 17 19:31:18.607889 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.607858    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776e39e9-1624-4974-8202-86caec2dbfbd" containerName="revoke-keys"
Jul 17 19:31:18.607889 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.607863    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="776e39e9-1624-4974-8202-86caec2dbfbd" containerName="revoke-keys"
Jul 17 19:31:18.607889 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.607878    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61afdfb6-a6fa-4461-8a2e-1494935a554d" containerName="main"
Jul 17 19:31:18.607889 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.607884    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="61afdfb6-a6fa-4461-8a2e-1494935a554d" containerName="main"
Jul 17 19:31:18.608047 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.607946    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="61afdfb6-a6fa-4461-8a2e-1494935a554d" containerName="main"
Jul 17 19:31:18.608047 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.607954    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7eaedd4-df54-44fa-9410-815a63230e10" containerName="revoke-keys"
Jul 17 19:31:18.611110 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.611090    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:18.613981 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.613724    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-98e305\"/\"default-dockercfg-5qjtz\""
Jul 17 19:31:18.613981 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.613794    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-98e305\"/\"openshift-service-ca.crt\""
Jul 17 19:31:18.613981 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.613823    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-98e305\"/\"kube-root-ca.crt\""
Jul 17 19:31:18.614198 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.614113    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-98e305\"/\"rate-test-model-98e305-kserve-self-signed-certs\""
Jul 17 19:31:18.622193 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.622173    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"]
Jul 17 19:31:18.680099 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.680005    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-home\") pod \"rate-test-model-98e305-kserve-85b7cd595f-hppzk\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") " pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:18.680099 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.680053    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-dshm\") pod \"rate-test-model-98e305-kserve-85b7cd595f-hppzk\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") " pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:18.680099 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.680073    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q6sv\" (UniqueName: \"kubernetes.io/projected/7d737418-1fcb-439e-be0b-35ec5020616e-kube-api-access-5q6sv\") pod \"rate-test-model-98e305-kserve-85b7cd595f-hppzk\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") " pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:18.680366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.680115    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-model-cache\") pod \"rate-test-model-98e305-kserve-85b7cd595f-hppzk\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") " pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:18.680366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.680152    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d737418-1fcb-439e-be0b-35ec5020616e-tls-certs\") pod \"rate-test-model-98e305-kserve-85b7cd595f-hppzk\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") " pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:18.680366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.680246    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-tmp-dir\") pod \"rate-test-model-98e305-kserve-85b7cd595f-hppzk\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") " pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:18.746865 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.746836    2559 generic.go:356] "Generic (PLEG): container finished" podID="61afdfb6-a6fa-4461-8a2e-1494935a554d" containerID="c074fc165ddf6011a2b0566477886e2ee9bee5e00f69a50768c667cb815682b3" exitCode=0
Jul 17 19:31:18.747002 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.746882    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"
Jul 17 19:31:18.747002 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.746909    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5" event={"ID":"61afdfb6-a6fa-4461-8a2e-1494935a554d","Type":"ContainerDied","Data":"c074fc165ddf6011a2b0566477886e2ee9bee5e00f69a50768c667cb815682b3"}
Jul 17 19:31:18.747002 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.746939    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5" event={"ID":"61afdfb6-a6fa-4461-8a2e-1494935a554d","Type":"ContainerDied","Data":"6c9ab78d0a648289f2d190d9ccb398c0d7d66915ed265d91618f6bef6e49999a"}
Jul 17 19:31:18.747002 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.746955    2559 scope.go:117] "RemoveContainer" containerID="c074fc165ddf6011a2b0566477886e2ee9bee5e00f69a50768c667cb815682b3"
Jul 17 19:31:18.756721 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.756699    2559 scope.go:117] "RemoveContainer" containerID="c074fc165ddf6011a2b0566477886e2ee9bee5e00f69a50768c667cb815682b3"
Jul 17 19:31:18.756975 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:31:18.756938    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c074fc165ddf6011a2b0566477886e2ee9bee5e00f69a50768c667cb815682b3\": container with ID starting with c074fc165ddf6011a2b0566477886e2ee9bee5e00f69a50768c667cb815682b3 not found: ID does not exist" containerID="c074fc165ddf6011a2b0566477886e2ee9bee5e00f69a50768c667cb815682b3"
Jul 17 19:31:18.757055 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.756980    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c074fc165ddf6011a2b0566477886e2ee9bee5e00f69a50768c667cb815682b3"} err="failed to get container status \"c074fc165ddf6011a2b0566477886e2ee9bee5e00f69a50768c667cb815682b3\": rpc error: code = NotFound desc = could not find container \"c074fc165ddf6011a2b0566477886e2ee9bee5e00f69a50768c667cb815682b3\": container with ID starting with c074fc165ddf6011a2b0566477886e2ee9bee5e00f69a50768c667cb815682b3 not found: ID does not exist"
Jul 17 19:31:18.768589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.768564    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"]
Jul 17 19:31:18.772421 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.772400    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-2109fd/sub-test-model-2109fd-kserve-65b78b775-pbrn5"]
Jul 17 19:31:18.781239 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.781222    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-tmp-dir\") pod \"rate-test-model-98e305-kserve-85b7cd595f-hppzk\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") " pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:18.781347 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.781255    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-home\") pod \"rate-test-model-98e305-kserve-85b7cd595f-hppzk\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") " pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:18.781420 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.781385    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-dshm\") pod \"rate-test-model-98e305-kserve-85b7cd595f-hppzk\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") " pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:18.781482 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.781422    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q6sv\" (UniqueName: \"kubernetes.io/projected/7d737418-1fcb-439e-be0b-35ec5020616e-kube-api-access-5q6sv\") pod \"rate-test-model-98e305-kserve-85b7cd595f-hppzk\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") " pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:18.781482 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.781450    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-model-cache\") pod \"rate-test-model-98e305-kserve-85b7cd595f-hppzk\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") " pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:18.781582 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.781481    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d737418-1fcb-439e-be0b-35ec5020616e-tls-certs\") pod \"rate-test-model-98e305-kserve-85b7cd595f-hppzk\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") " pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:18.781582 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.781548    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-home\") pod \"rate-test-model-98e305-kserve-85b7cd595f-hppzk\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") " pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:18.781683 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.781612    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-tmp-dir\") pod \"rate-test-model-98e305-kserve-85b7cd595f-hppzk\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") " pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:18.781745 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.781733    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-model-cache\") pod \"rate-test-model-98e305-kserve-85b7cd595f-hppzk\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") " pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:18.783598 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.783580    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-dshm\") pod \"rate-test-model-98e305-kserve-85b7cd595f-hppzk\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") " pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:18.783918 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.783898    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d737418-1fcb-439e-be0b-35ec5020616e-tls-certs\") pod \"rate-test-model-98e305-kserve-85b7cd595f-hppzk\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") " pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:18.790403 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.790384    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q6sv\" (UniqueName: \"kubernetes.io/projected/7d737418-1fcb-439e-be0b-35ec5020616e-kube-api-access-5q6sv\") pod \"rate-test-model-98e305-kserve-85b7cd595f-hppzk\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") " pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:18.923326 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:18.923285    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:19.065346 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:19.065301    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"]
Jul 17 19:31:19.065957 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:31:19.065933    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d737418_1fcb_439e_be0b_35ec5020616e.slice/crio-34bededdd460703bef0f0ee931b9d122f1fda9dbe04876e42f5b6d4c7f397590 WatchSource:0}: Error finding container 34bededdd460703bef0f0ee931b9d122f1fda9dbe04876e42f5b6d4c7f397590: Status 404 returned error can't find the container with id 34bededdd460703bef0f0ee931b9d122f1fda9dbe04876e42f5b6d4c7f397590
Jul 17 19:31:19.755521 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:19.755481    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk" event={"ID":"7d737418-1fcb-439e-be0b-35ec5020616e","Type":"ContainerStarted","Data":"c479c4d0023a1b37155fc5ef0c1bb466691d544ffc62c06ecf54027ff869937e"}
Jul 17 19:31:19.755521 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:19.755526    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk" event={"ID":"7d737418-1fcb-439e-be0b-35ec5020616e","Type":"ContainerStarted","Data":"34bededdd460703bef0f0ee931b9d122f1fda9dbe04876e42f5b6d4c7f397590"}
Jul 17 19:31:19.781327 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:19.777936    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk" podStartSLOduration=1.7779169860000001 podStartE2EDuration="1.777916986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:31:19.773741141 +0000 UTC m=+3490.215249829" watchObservedRunningTime="2026-07-17 19:31:19.777916986 +0000 UTC m=+3490.219425676"
Jul 17 19:31:20.125626 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:20.125589    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61afdfb6-a6fa-4461-8a2e-1494935a554d" path="/var/lib/kubelet/pods/61afdfb6-a6fa-4461-8a2e-1494935a554d/volumes"
Jul 17 19:31:28.924259 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:28.924222    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:28.924760 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:28.924280    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:28.936773 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:28.936749    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:29.119270 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:29.119239    2559 scope.go:117] "RemoveContainer" containerID="74ce38170f4d286f90bd21210d417eac806d6ed0e4117116545b79f3050f4644"
Jul 17 19:31:29.816466 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:29.816432    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5" event={"ID":"c542e7a9-8812-433b-a6e1-d72d204abdfe","Type":"ContainerStarted","Data":"b80d2e1220c9819ec6f07ea55145501194e2f69b827a1029d424c1281e6649a9"}
Jul 17 19:31:29.828145 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:29.828117    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:31:29.832941 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:29.832890    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5" podStartSLOduration=16.832875872 podStartE2EDuration="16.832875872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:31:29.832127174 +0000 UTC m=+3500.273635854" watchObservedRunningTime="2026-07-17 19:31:29.832875872 +0000 UTC m=+3500.274384561"
Jul 17 19:31:30.824344 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:30.824293    2559 generic.go:356] "Generic (PLEG): container finished" podID="c542e7a9-8812-433b-a6e1-d72d204abdfe" containerID="b80d2e1220c9819ec6f07ea55145501194e2f69b827a1029d424c1281e6649a9" exitCode=6
Jul 17 19:31:30.824795 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:30.824368    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5" event={"ID":"c542e7a9-8812-433b-a6e1-d72d204abdfe","Type":"ContainerDied","Data":"b80d2e1220c9819ec6f07ea55145501194e2f69b827a1029d424c1281e6649a9"}
Jul 17 19:31:30.824795 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:30.824421    2559 scope.go:117] "RemoveContainer" containerID="74ce38170f4d286f90bd21210d417eac806d6ed0e4117116545b79f3050f4644"
Jul 17 19:31:30.824795 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:30.824777    2559 scope.go:117] "RemoveContainer" containerID="b80d2e1220c9819ec6f07ea55145501194e2f69b827a1029d424c1281e6649a9"
Jul 17 19:31:30.824955 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:31:30.824936    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5_odh-ai-gateway-infra(c542e7a9-8812-433b-a6e1-d72d204abdfe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5" podUID="c542e7a9-8812-433b-a6e1-d72d204abdfe"
Jul 17 19:31:30.850183 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:30.850155    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5"]
Jul 17 19:31:31.968750 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:31.968723    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5"
Jul 17 19:31:32.098963 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:32.098880    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c542e7a9-8812-433b-a6e1-d72d204abdfe-maas-api-service-ca\") pod \"c542e7a9-8812-433b-a6e1-d72d204abdfe\" (UID: \"c542e7a9-8812-433b-a6e1-d72d204abdfe\") "
Jul 17 19:31:32.099228 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:32.099205    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c542e7a9-8812-433b-a6e1-d72d204abdfe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c542e7a9-8812-433b-a6e1-d72d204abdfe" (UID: "c542e7a9-8812-433b-a6e1-d72d204abdfe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:31:32.200419 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:32.200386    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c542e7a9-8812-433b-a6e1-d72d204abdfe-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:31:32.838500 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:32.838470    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5"
Jul 17 19:31:32.838693 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:32.838512    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5" event={"ID":"c542e7a9-8812-433b-a6e1-d72d204abdfe","Type":"ContainerDied","Data":"6b5c8550f013b8cc144692bf766671507ddb8dce9acf5ca5f25bff9d33d71fe9"}
Jul 17 19:31:32.838693 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:32.838577    2559 scope.go:117] "RemoveContainer" containerID="b80d2e1220c9819ec6f07ea55145501194e2f69b827a1029d424c1281e6649a9"
Jul 17 19:31:32.857439 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:32.857418    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5"]
Jul 17 19:31:32.861001 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:32.860975    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-j7hf5"]
Jul 17 19:31:34.126125 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:34.126090    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c542e7a9-8812-433b-a6e1-d72d204abdfe" path="/var/lib/kubelet/pods/c542e7a9-8812-433b-a6e1-d72d204abdfe/volumes"
Jul 17 19:31:41.646831 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.646799    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67"]
Jul 17 19:31:41.647240 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.647223    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c542e7a9-8812-433b-a6e1-d72d204abdfe" containerName="revoke-keys"
Jul 17 19:31:41.647240 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.647234    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c542e7a9-8812-433b-a6e1-d72d204abdfe" containerName="revoke-keys"
Jul 17 19:31:41.647240 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.647241    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c542e7a9-8812-433b-a6e1-d72d204abdfe" containerName="revoke-keys"
Jul 17 19:31:41.647387 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.647247    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c542e7a9-8812-433b-a6e1-d72d204abdfe" containerName="revoke-keys"
Jul 17 19:31:41.647387 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.647347    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c542e7a9-8812-433b-a6e1-d72d204abdfe" containerName="revoke-keys"
Jul 17 19:31:41.647387 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.647357    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c542e7a9-8812-433b-a6e1-d72d204abdfe" containerName="revoke-keys"
Jul 17 19:31:41.647387 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.647363    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c542e7a9-8812-433b-a6e1-d72d204abdfe" containerName="revoke-keys"
Jul 17 19:31:41.651972 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.651956    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67"
Jul 17 19:31:41.654961 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.654936    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:31:41.658117 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.658094    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67"]
Jul 17 19:31:41.786215 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.786184    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f0291f2-b2ce-470d-b219-bd23bfdc8e6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-n2m67\" (UID: \"8f0291f2-b2ce-470d-b219-bd23bfdc8e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67"
Jul 17 19:31:41.805831 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.805804    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"]
Jul 17 19:31:41.806239 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.806227    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c542e7a9-8812-433b-a6e1-d72d204abdfe" containerName="revoke-keys"
Jul 17 19:31:41.806239 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.806240    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c542e7a9-8812-433b-a6e1-d72d204abdfe" containerName="revoke-keys"
Jul 17 19:31:41.809575 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.809552    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:41.812372 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.812352    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2109fd\"/\"default-dockercfg-zf2rg\""
Jul 17 19:31:41.812495 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.812452    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2109fd\"/\"openshift-service-ca.crt\""
Jul 17 19:31:41.812566 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.812502    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2109fd\"/\"kube-root-ca.crt\""
Jul 17 19:31:41.812566 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.812515    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2109fd\"/\"rate-test-model-2109fd-kserve-self-signed-certs\""
Jul 17 19:31:41.819324 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.819281    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"]
Jul 17 19:31:41.887380 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.887357    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-tmp-dir\") pod \"rate-test-model-2109fd-kserve-6df5445575-9gt4w\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") " pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:41.887546 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.887429    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-dshm\") pod \"rate-test-model-2109fd-kserve-6df5445575-9gt4w\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") " pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:41.887546 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.887470    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f0291f2-b2ce-470d-b219-bd23bfdc8e6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-n2m67\" (UID: \"8f0291f2-b2ce-470d-b219-bd23bfdc8e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67"
Jul 17 19:31:41.887546 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.887500    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-home\") pod \"rate-test-model-2109fd-kserve-6df5445575-9gt4w\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") " pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:41.887725 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.887584    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-model-cache\") pod \"rate-test-model-2109fd-kserve-6df5445575-9gt4w\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") " pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:41.887725 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.887619    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkx2n\" (UniqueName: \"kubernetes.io/projected/f3eb86c7-e264-4da9-9992-d80b9d3051f7-kube-api-access-lkx2n\") pod \"rate-test-model-2109fd-kserve-6df5445575-9gt4w\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") " pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:41.887725 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.887687    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3eb86c7-e264-4da9-9992-d80b9d3051f7-tls-certs\") pod \"rate-test-model-2109fd-kserve-6df5445575-9gt4w\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") " pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:41.888057 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.888039    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f0291f2-b2ce-470d-b219-bd23bfdc8e6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-n2m67\" (UID: \"8f0291f2-b2ce-470d-b219-bd23bfdc8e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67"
Jul 17 19:31:41.963830 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.963770    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67"
Jul 17 19:31:41.988919 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.988896    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-tmp-dir\") pod \"rate-test-model-2109fd-kserve-6df5445575-9gt4w\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") " pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:41.989023 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.988953    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-dshm\") pod \"rate-test-model-2109fd-kserve-6df5445575-9gt4w\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") " pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:41.989023 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.988987    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-home\") pod \"rate-test-model-2109fd-kserve-6df5445575-9gt4w\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") " pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:41.989023 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.989012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-model-cache\") pod \"rate-test-model-2109fd-kserve-6df5445575-9gt4w\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") " pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:41.989189 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.989028    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkx2n\" (UniqueName: \"kubernetes.io/projected/f3eb86c7-e264-4da9-9992-d80b9d3051f7-kube-api-access-lkx2n\") pod \"rate-test-model-2109fd-kserve-6df5445575-9gt4w\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") " pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:41.989189 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.989060    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3eb86c7-e264-4da9-9992-d80b9d3051f7-tls-certs\") pod \"rate-test-model-2109fd-kserve-6df5445575-9gt4w\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") " pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:41.989364 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.989343    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-tmp-dir\") pod \"rate-test-model-2109fd-kserve-6df5445575-9gt4w\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") " pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:41.989443 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.989362    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-model-cache\") pod \"rate-test-model-2109fd-kserve-6df5445575-9gt4w\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") " pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:41.989568 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.989550    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-home\") pod \"rate-test-model-2109fd-kserve-6df5445575-9gt4w\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") " pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:41.991580 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.991560    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-dshm\") pod \"rate-test-model-2109fd-kserve-6df5445575-9gt4w\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") " pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:41.991717 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.991701    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3eb86c7-e264-4da9-9992-d80b9d3051f7-tls-certs\") pod \"rate-test-model-2109fd-kserve-6df5445575-9gt4w\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") " pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:41.998409 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:41.998388    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkx2n\" (UniqueName: \"kubernetes.io/projected/f3eb86c7-e264-4da9-9992-d80b9d3051f7-kube-api-access-lkx2n\") pod \"rate-test-model-2109fd-kserve-6df5445575-9gt4w\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") " pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:42.108505 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:42.106761    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67"]
Jul 17 19:31:42.109150 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:31:42.109125    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0291f2_b2ce_470d_b219_bd23bfdc8e6d.slice/crio-937980d8ec78cfda907579ae904cf9a782403fc20dc86bf0c46ddfae26c0c9ac WatchSource:0}: Error finding container 937980d8ec78cfda907579ae904cf9a782403fc20dc86bf0c46ddfae26c0c9ac: Status 404 returned error can't find the container with id 937980d8ec78cfda907579ae904cf9a782403fc20dc86bf0c46ddfae26c0c9ac
Jul 17 19:31:42.119946 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:42.119909    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:42.253267 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:42.253236    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"]
Jul 17 19:31:42.254394 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:31:42.254370    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3eb86c7_e264_4da9_9992_d80b9d3051f7.slice/crio-f1e0558f368f05d39c546b52412e1a98069dff07bc737ce984379d0a27f50471 WatchSource:0}: Error finding container f1e0558f368f05d39c546b52412e1a98069dff07bc737ce984379d0a27f50471: Status 404 returned error can't find the container with id f1e0558f368f05d39c546b52412e1a98069dff07bc737ce984379d0a27f50471
Jul 17 19:31:42.896342 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:42.896284    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w" event={"ID":"f3eb86c7-e264-4da9-9992-d80b9d3051f7","Type":"ContainerStarted","Data":"c636de237090c6f13130b993fc4a6b688922b361692e4956d88c0e6c143f8448"}
Jul 17 19:31:42.896760 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:42.896358    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w" event={"ID":"f3eb86c7-e264-4da9-9992-d80b9d3051f7","Type":"ContainerStarted","Data":"f1e0558f368f05d39c546b52412e1a98069dff07bc737ce984379d0a27f50471"}
Jul 17 19:31:42.897843 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:42.897818    2559 generic.go:356] "Generic (PLEG): container finished" podID="8f0291f2-b2ce-470d-b219-bd23bfdc8e6d" containerID="3ab838cf67c65376cf3fda198ca76d7f04bf7985185cf2544588b2570a9ae391" exitCode=6
Jul 17 19:31:42.897991 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:42.897884    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67" event={"ID":"8f0291f2-b2ce-470d-b219-bd23bfdc8e6d","Type":"ContainerDied","Data":"3ab838cf67c65376cf3fda198ca76d7f04bf7985185cf2544588b2570a9ae391"}
Jul 17 19:31:42.897991 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:42.897941    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67" event={"ID":"8f0291f2-b2ce-470d-b219-bd23bfdc8e6d","Type":"ContainerStarted","Data":"937980d8ec78cfda907579ae904cf9a782403fc20dc86bf0c46ddfae26c0c9ac"}
Jul 17 19:31:42.898086 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:42.898015    2559 scope.go:117] "RemoveContainer" containerID="3ab838cf67c65376cf3fda198ca76d7f04bf7985185cf2544588b2570a9ae391"
Jul 17 19:31:42.916267 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:42.916226    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w" podStartSLOduration=1.9162113600000001 podStartE2EDuration="1.91621136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:31:42.913461033 +0000 UTC m=+3513.354969723" watchObservedRunningTime="2026-07-17 19:31:42.91621136 +0000 UTC m=+3513.357720050"
Jul 17 19:31:43.905918 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:43.905880    2559 generic.go:356] "Generic (PLEG): container finished" podID="8f0291f2-b2ce-470d-b219-bd23bfdc8e6d" containerID="5672c3df624a6ceec97d61e7a12c9c796c84f54943ed242fd1efa4d4d8d7be7a" exitCode=6
Jul 17 19:31:43.906377 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:43.905959    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67" event={"ID":"8f0291f2-b2ce-470d-b219-bd23bfdc8e6d","Type":"ContainerDied","Data":"5672c3df624a6ceec97d61e7a12c9c796c84f54943ed242fd1efa4d4d8d7be7a"}
Jul 17 19:31:43.906377 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:43.906015    2559 scope.go:117] "RemoveContainer" containerID="3ab838cf67c65376cf3fda198ca76d7f04bf7985185cf2544588b2570a9ae391"
Jul 17 19:31:43.906377 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:43.906168    2559 scope.go:117] "RemoveContainer" containerID="5672c3df624a6ceec97d61e7a12c9c796c84f54943ed242fd1efa4d4d8d7be7a"
Jul 17 19:31:43.906516 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:31:43.906438    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-n2m67_odh-ai-gateway-infra(8f0291f2-b2ce-470d-b219-bd23bfdc8e6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67" podUID="8f0291f2-b2ce-470d-b219-bd23bfdc8e6d"
Jul 17 19:31:44.535188 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:44.535158    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj"]
Jul 17 19:31:44.540232 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:44.540216    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj"
Jul 17 19:31:44.547581 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:44.547555    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj"]
Jul 17 19:31:44.613915 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:44.613885    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/105a5e13-7e1d-41f6-b065-677b3e6c8523-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj\" (UID: \"105a5e13-7e1d-41f6-b065-677b3e6c8523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj"
Jul 17 19:31:44.714656 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:44.714628    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/105a5e13-7e1d-41f6-b065-677b3e6c8523-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj\" (UID: \"105a5e13-7e1d-41f6-b065-677b3e6c8523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj"
Jul 17 19:31:44.715231 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:44.715213    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/105a5e13-7e1d-41f6-b065-677b3e6c8523-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj\" (UID: \"105a5e13-7e1d-41f6-b065-677b3e6c8523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj"
Jul 17 19:31:44.852150 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:44.852127    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj"
Jul 17 19:31:44.918155 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:44.918122    2559 scope.go:117] "RemoveContainer" containerID="5672c3df624a6ceec97d61e7a12c9c796c84f54943ed242fd1efa4d4d8d7be7a"
Jul 17 19:31:44.918598 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:31:44.918380    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-n2m67_odh-ai-gateway-infra(8f0291f2-b2ce-470d-b219-bd23bfdc8e6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67" podUID="8f0291f2-b2ce-470d-b219-bd23bfdc8e6d"
Jul 17 19:31:44.993555 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:44.993525    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj"]
Jul 17 19:31:44.993759 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:31:44.993733    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105a5e13_7e1d_41f6_b065_677b3e6c8523.slice/crio-8ec92471e818e3bb1b756c1bc09c9f3fdb83fafefaec6a29752b597f3eb6ed61 WatchSource:0}: Error finding container 8ec92471e818e3bb1b756c1bc09c9f3fdb83fafefaec6a29752b597f3eb6ed61: Status 404 returned error can't find the container with id 8ec92471e818e3bb1b756c1bc09c9f3fdb83fafefaec6a29752b597f3eb6ed61
Jul 17 19:31:45.926656 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:45.926575    2559 generic.go:356] "Generic (PLEG): container finished" podID="105a5e13-7e1d-41f6-b065-677b3e6c8523" containerID="e41c39e15e545abe49203b3d3c4dec392aef689e6113749b0e8adcb72e32f595" exitCode=6
Jul 17 19:31:45.927011 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:45.926663    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj" event={"ID":"105a5e13-7e1d-41f6-b065-677b3e6c8523","Type":"ContainerDied","Data":"e41c39e15e545abe49203b3d3c4dec392aef689e6113749b0e8adcb72e32f595"}
Jul 17 19:31:45.927011 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:45.926752    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj" event={"ID":"105a5e13-7e1d-41f6-b065-677b3e6c8523","Type":"ContainerStarted","Data":"8ec92471e818e3bb1b756c1bc09c9f3fdb83fafefaec6a29752b597f3eb6ed61"}
Jul 17 19:31:45.927011 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:45.926811    2559 scope.go:117] "RemoveContainer" containerID="e41c39e15e545abe49203b3d3c4dec392aef689e6113749b0e8adcb72e32f595"
Jul 17 19:31:46.934889 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:46.934847    2559 generic.go:356] "Generic (PLEG): container finished" podID="105a5e13-7e1d-41f6-b065-677b3e6c8523" containerID="7209dcbde5435ad450fa051a1c7d073a871ebb969364abfdd27edef9a81674fa" exitCode=6
Jul 17 19:31:46.935401 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:46.934937    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj" event={"ID":"105a5e13-7e1d-41f6-b065-677b3e6c8523","Type":"ContainerDied","Data":"7209dcbde5435ad450fa051a1c7d073a871ebb969364abfdd27edef9a81674fa"}
Jul 17 19:31:46.935401 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:46.934993    2559 scope.go:117] "RemoveContainer" containerID="e41c39e15e545abe49203b3d3c4dec392aef689e6113749b0e8adcb72e32f595"
Jul 17 19:31:46.935401 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:46.935250    2559 scope.go:117] "RemoveContainer" containerID="7209dcbde5435ad450fa051a1c7d073a871ebb969364abfdd27edef9a81674fa"
Jul 17 19:31:46.935573 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:31:46.935538    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj_odh-ai-gateway-infra(105a5e13-7e1d-41f6-b065-677b3e6c8523)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj" podUID="105a5e13-7e1d-41f6-b065-677b3e6c8523"
Jul 17 19:31:47.942742 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:47.942719    2559 scope.go:117] "RemoveContainer" containerID="7209dcbde5435ad450fa051a1c7d073a871ebb969364abfdd27edef9a81674fa"
Jul 17 19:31:47.943124 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:31:47.942898    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj_odh-ai-gateway-infra(105a5e13-7e1d-41f6-b065-677b3e6c8523)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj" podUID="105a5e13-7e1d-41f6-b065-677b3e6c8523"
Jul 17 19:31:52.134611 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:52.134583    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:52.135000 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:52.134623    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:52.136479 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:52.136458    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:52.984200 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:52.984170    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:31:57.120081 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:57.120056    2559 scope.go:117] "RemoveContainer" containerID="5672c3df624a6ceec97d61e7a12c9c796c84f54943ed242fd1efa4d4d8d7be7a"
Jul 17 19:31:58.002576 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:58.002543    2559 generic.go:356] "Generic (PLEG): container finished" podID="8f0291f2-b2ce-470d-b219-bd23bfdc8e6d" containerID="05de70c7aff67cd644299337e8ddc25ff49df9cadb8dc40d364dd3e97da989ab" exitCode=6
Jul 17 19:31:58.002784 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:58.002623    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67" event={"ID":"8f0291f2-b2ce-470d-b219-bd23bfdc8e6d","Type":"ContainerDied","Data":"05de70c7aff67cd644299337e8ddc25ff49df9cadb8dc40d364dd3e97da989ab"}
Jul 17 19:31:58.002784 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:58.002694    2559 scope.go:117] "RemoveContainer" containerID="5672c3df624a6ceec97d61e7a12c9c796c84f54943ed242fd1efa4d4d8d7be7a"
Jul 17 19:31:58.002988 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:58.002969    2559 scope.go:117] "RemoveContainer" containerID="05de70c7aff67cd644299337e8ddc25ff49df9cadb8dc40d364dd3e97da989ab"
Jul 17 19:31:58.003235 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:31:58.003217    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-n2m67_odh-ai-gateway-infra(8f0291f2-b2ce-470d-b219-bd23bfdc8e6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67" podUID="8f0291f2-b2ce-470d-b219-bd23bfdc8e6d"
Jul 17 19:31:59.030763 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:59.030731    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67"]
Jul 17 19:31:59.168995 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:59.168971    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67"
Jul 17 19:31:59.344916 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:59.344882    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f0291f2-b2ce-470d-b219-bd23bfdc8e6d-maas-api-service-ca\") pod \"8f0291f2-b2ce-470d-b219-bd23bfdc8e6d\" (UID: \"8f0291f2-b2ce-470d-b219-bd23bfdc8e6d\") "
Jul 17 19:31:59.345276 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:59.345253    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0291f2-b2ce-470d-b219-bd23bfdc8e6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f0291f2-b2ce-470d-b219-bd23bfdc8e6d" (UID: "8f0291f2-b2ce-470d-b219-bd23bfdc8e6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:31:59.446555 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:31:59.446527    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f0291f2-b2ce-470d-b219-bd23bfdc8e6d-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:00.017485 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:00.017450    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67"
Jul 17 19:32:00.017670 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:00.017489    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67" event={"ID":"8f0291f2-b2ce-470d-b219-bd23bfdc8e6d","Type":"ContainerDied","Data":"937980d8ec78cfda907579ae904cf9a782403fc20dc86bf0c46ddfae26c0c9ac"}
Jul 17 19:32:00.017670 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:00.017582    2559 scope.go:117] "RemoveContainer" containerID="05de70c7aff67cd644299337e8ddc25ff49df9cadb8dc40d364dd3e97da989ab"
Jul 17 19:32:00.039286 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:00.039260    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67"]
Jul 17 19:32:00.042488 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:00.042467    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-n2m67"]
Jul 17 19:32:00.126176 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:00.126132    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f0291f2-b2ce-470d-b219-bd23bfdc8e6d" path="/var/lib/kubelet/pods/8f0291f2-b2ce-470d-b219-bd23bfdc8e6d/volumes"
Jul 17 19:32:02.120341 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:02.120289    2559 scope.go:117] "RemoveContainer" containerID="7209dcbde5435ad450fa051a1c7d073a871ebb969364abfdd27edef9a81674fa"
Jul 17 19:32:03.038359 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:03.038263    2559 generic.go:356] "Generic (PLEG): container finished" podID="105a5e13-7e1d-41f6-b065-677b3e6c8523" containerID="63589335be6c4a3a8556ac1ddac9d14f7ec38c12424aeab8aeccc1eab2929cc9" exitCode=6
Jul 17 19:32:03.038359 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:03.038344    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj" event={"ID":"105a5e13-7e1d-41f6-b065-677b3e6c8523","Type":"ContainerDied","Data":"63589335be6c4a3a8556ac1ddac9d14f7ec38c12424aeab8aeccc1eab2929cc9"}
Jul 17 19:32:03.038549 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:03.038407    2559 scope.go:117] "RemoveContainer" containerID="7209dcbde5435ad450fa051a1c7d073a871ebb969364abfdd27edef9a81674fa"
Jul 17 19:32:03.038636 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:03.038623    2559 scope.go:117] "RemoveContainer" containerID="63589335be6c4a3a8556ac1ddac9d14f7ec38c12424aeab8aeccc1eab2929cc9"
Jul 17 19:32:03.038872 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:32:03.038848    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj_odh-ai-gateway-infra(105a5e13-7e1d-41f6-b065-677b3e6c8523)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj" podUID="105a5e13-7e1d-41f6-b065-677b3e6c8523"
Jul 17 19:32:03.424395 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:03.424365    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95"]
Jul 17 19:32:03.424751 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:03.424737    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f0291f2-b2ce-470d-b219-bd23bfdc8e6d" containerName="revoke-keys"
Jul 17 19:32:03.424751 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:03.424748    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f0291f2-b2ce-470d-b219-bd23bfdc8e6d" containerName="revoke-keys"
Jul 17 19:32:03.424848 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:03.424761    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f0291f2-b2ce-470d-b219-bd23bfdc8e6d" containerName="revoke-keys"
Jul 17 19:32:03.424848 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:03.424767    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f0291f2-b2ce-470d-b219-bd23bfdc8e6d" containerName="revoke-keys"
Jul 17 19:32:03.424848 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:03.424776    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f0291f2-b2ce-470d-b219-bd23bfdc8e6d" containerName="revoke-keys"
Jul 17 19:32:03.424848 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:03.424781    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f0291f2-b2ce-470d-b219-bd23bfdc8e6d" containerName="revoke-keys"
Jul 17 19:32:03.424848 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:03.424849    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f0291f2-b2ce-470d-b219-bd23bfdc8e6d" containerName="revoke-keys"
Jul 17 19:32:03.425009 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:03.424857    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f0291f2-b2ce-470d-b219-bd23bfdc8e6d" containerName="revoke-keys"
Jul 17 19:32:03.429143 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:03.429125    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95"
Jul 17 19:32:03.434779 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:03.434756    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95"]
Jul 17 19:32:03.584863 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:03.584833    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f371021-df80-4915-b8fe-13ebc1656423-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95\" (UID: \"8f371021-df80-4915-b8fe-13ebc1656423\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95"
Jul 17 19:32:03.685749 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:03.685662    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f371021-df80-4915-b8fe-13ebc1656423-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95\" (UID: \"8f371021-df80-4915-b8fe-13ebc1656423\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95"
Jul 17 19:32:03.686260 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:03.686240    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f371021-df80-4915-b8fe-13ebc1656423-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95\" (UID: \"8f371021-df80-4915-b8fe-13ebc1656423\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95"
Jul 17 19:32:03.741530 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:03.741504    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95"
Jul 17 19:32:03.887544 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:03.886256    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95"]
Jul 17 19:32:03.889244 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:32:03.889208    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f371021_df80_4915_b8fe_13ebc1656423.slice/crio-b59d58ec45ec022ac2c442d4541df023115739e8b0aca5dc81fcd4fce17ae537 WatchSource:0}: Error finding container b59d58ec45ec022ac2c442d4541df023115739e8b0aca5dc81fcd4fce17ae537: Status 404 returned error can't find the container with id b59d58ec45ec022ac2c442d4541df023115739e8b0aca5dc81fcd4fce17ae537
Jul 17 19:32:04.054060 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:04.054029    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95" event={"ID":"8f371021-df80-4915-b8fe-13ebc1656423","Type":"ContainerStarted","Data":"b59d58ec45ec022ac2c442d4541df023115739e8b0aca5dc81fcd4fce17ae537"}
Jul 17 19:32:04.068647 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:04.068623    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj"]
Jul 17 19:32:04.221092 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:04.221067    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj"
Jul 17 19:32:04.292906 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:04.292880    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/105a5e13-7e1d-41f6-b065-677b3e6c8523-maas-api-service-ca\") pod \"105a5e13-7e1d-41f6-b065-677b3e6c8523\" (UID: \"105a5e13-7e1d-41f6-b065-677b3e6c8523\") "
Jul 17 19:32:04.293204 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:04.293183    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105a5e13-7e1d-41f6-b065-677b3e6c8523-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "105a5e13-7e1d-41f6-b065-677b3e6c8523" (UID: "105a5e13-7e1d-41f6-b065-677b3e6c8523"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:32:04.394001 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:04.393961    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/105a5e13-7e1d-41f6-b065-677b3e6c8523-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:05.062065 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:05.062030    2559 generic.go:356] "Generic (PLEG): container finished" podID="8f371021-df80-4915-b8fe-13ebc1656423" containerID="6c29f1ed18dcdfd687c1395bd4103430bef025b6cea049d61c73c32a646a1913" exitCode=6
Jul 17 19:32:05.062706 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:05.062075    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95" event={"ID":"8f371021-df80-4915-b8fe-13ebc1656423","Type":"ContainerDied","Data":"6c29f1ed18dcdfd687c1395bd4103430bef025b6cea049d61c73c32a646a1913"}
Jul 17 19:32:05.062706 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:05.062350    2559 scope.go:117] "RemoveContainer" containerID="6c29f1ed18dcdfd687c1395bd4103430bef025b6cea049d61c73c32a646a1913"
Jul 17 19:32:05.063704 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:05.063683    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj"
Jul 17 19:32:05.063805 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:05.063709    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj" event={"ID":"105a5e13-7e1d-41f6-b065-677b3e6c8523","Type":"ContainerDied","Data":"8ec92471e818e3bb1b756c1bc09c9f3fdb83fafefaec6a29752b597f3eb6ed61"}
Jul 17 19:32:05.063805 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:05.063752    2559 scope.go:117] "RemoveContainer" containerID="63589335be6c4a3a8556ac1ddac9d14f7ec38c12424aeab8aeccc1eab2929cc9"
Jul 17 19:32:05.093178 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:05.093147    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj"]
Jul 17 19:32:05.096583 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:05.096563    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-pmqsj"]
Jul 17 19:32:06.071703 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:06.071671    2559 generic.go:356] "Generic (PLEG): container finished" podID="8f371021-df80-4915-b8fe-13ebc1656423" containerID="7cf40db56263a82399d7ef787b462c3a25e61989612b89a29416d283d7baf033" exitCode=6
Jul 17 19:32:06.072170 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:06.071754    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95" event={"ID":"8f371021-df80-4915-b8fe-13ebc1656423","Type":"ContainerDied","Data":"7cf40db56263a82399d7ef787b462c3a25e61989612b89a29416d283d7baf033"}
Jul 17 19:32:06.072170 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:06.071813    2559 scope.go:117] "RemoveContainer" containerID="6c29f1ed18dcdfd687c1395bd4103430bef025b6cea049d61c73c32a646a1913"
Jul 17 19:32:06.072170 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:06.072001    2559 scope.go:117] "RemoveContainer" containerID="7cf40db56263a82399d7ef787b462c3a25e61989612b89a29416d283d7baf033"
Jul 17 19:32:06.072455 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:32:06.072215    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95_odh-ai-gateway-infra(8f371021-df80-4915-b8fe-13ebc1656423)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95" podUID="8f371021-df80-4915-b8fe-13ebc1656423"
Jul 17 19:32:06.126268 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:06.126231    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="105a5e13-7e1d-41f6-b065-677b3e6c8523" path="/var/lib/kubelet/pods/105a5e13-7e1d-41f6-b065-677b3e6c8523/volumes"
Jul 17 19:32:07.080034 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:07.080007    2559 scope.go:117] "RemoveContainer" containerID="7cf40db56263a82399d7ef787b462c3a25e61989612b89a29416d283d7baf033"
Jul 17 19:32:07.080413 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:32:07.080170    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95_odh-ai-gateway-infra(8f371021-df80-4915-b8fe-13ebc1656423)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95" podUID="8f371021-df80-4915-b8fe-13ebc1656423"
Jul 17 19:32:18.120423 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:18.120390    2559 scope.go:117] "RemoveContainer" containerID="7cf40db56263a82399d7ef787b462c3a25e61989612b89a29416d283d7baf033"
Jul 17 19:32:19.156202 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:19.156172    2559 generic.go:356] "Generic (PLEG): container finished" podID="8f371021-df80-4915-b8fe-13ebc1656423" containerID="1acbea4541756b5deeff859dc45d9a14e948d963602099fc8e0432c46936b3c6" exitCode=6
Jul 17 19:32:19.156665 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:19.156245    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95" event={"ID":"8f371021-df80-4915-b8fe-13ebc1656423","Type":"ContainerDied","Data":"1acbea4541756b5deeff859dc45d9a14e948d963602099fc8e0432c46936b3c6"}
Jul 17 19:32:19.156665 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:19.156301    2559 scope.go:117] "RemoveContainer" containerID="7cf40db56263a82399d7ef787b462c3a25e61989612b89a29416d283d7baf033"
Jul 17 19:32:19.156665 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:19.156555    2559 scope.go:117] "RemoveContainer" containerID="1acbea4541756b5deeff859dc45d9a14e948d963602099fc8e0432c46936b3c6"
Jul 17 19:32:19.156798 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:32:19.156765    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95_odh-ai-gateway-infra(8f371021-df80-4915-b8fe-13ebc1656423)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95" podUID="8f371021-df80-4915-b8fe-13ebc1656423"
Jul 17 19:32:20.180852 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:20.180814    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95"]
Jul 17 19:32:20.322134 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:20.322110    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95"
Jul 17 19:32:20.417081 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:20.417058    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f371021-df80-4915-b8fe-13ebc1656423-maas-api-service-ca\") pod \"8f371021-df80-4915-b8fe-13ebc1656423\" (UID: \"8f371021-df80-4915-b8fe-13ebc1656423\") "
Jul 17 19:32:20.417468 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:20.417445    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f371021-df80-4915-b8fe-13ebc1656423-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f371021-df80-4915-b8fe-13ebc1656423" (UID: "8f371021-df80-4915-b8fe-13ebc1656423"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:32:20.518130 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:20.518068    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f371021-df80-4915-b8fe-13ebc1656423-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:21.170499 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:21.170469    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95"
Jul 17 19:32:21.170697 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:21.170512    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95" event={"ID":"8f371021-df80-4915-b8fe-13ebc1656423","Type":"ContainerDied","Data":"b59d58ec45ec022ac2c442d4541df023115739e8b0aca5dc81fcd4fce17ae537"}
Jul 17 19:32:21.170697 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:21.170568    2559 scope.go:117] "RemoveContainer" containerID="1acbea4541756b5deeff859dc45d9a14e948d963602099fc8e0432c46936b3c6"
Jul 17 19:32:21.193898 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:21.193872    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95"]
Jul 17 19:32:21.200615 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:21.200592    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-jcw95"]
Jul 17 19:32:22.126831 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:22.126800    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f371021-df80-4915-b8fe-13ebc1656423" path="/var/lib/kubelet/pods/8f371021-df80-4915-b8fe-13ebc1656423/volumes"
Jul 17 19:32:28.133212 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:28.133167    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"]
Jul 17 19:32:28.929499 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:28.929466    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"]
Jul 17 19:32:31.685667 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.685631    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j"]
Jul 17 19:32:31.686018 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.685986    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="105a5e13-7e1d-41f6-b065-677b3e6c8523" containerName="revoke-keys"
Jul 17 19:32:31.686018 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.685997    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="105a5e13-7e1d-41f6-b065-677b3e6c8523" containerName="revoke-keys"
Jul 17 19:32:31.686018 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.686012    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f371021-df80-4915-b8fe-13ebc1656423" containerName="revoke-keys"
Jul 17 19:32:31.686018 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.686018    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f371021-df80-4915-b8fe-13ebc1656423" containerName="revoke-keys"
Jul 17 19:32:31.686178 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.686024    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="105a5e13-7e1d-41f6-b065-677b3e6c8523" containerName="revoke-keys"
Jul 17 19:32:31.686178 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.686029    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="105a5e13-7e1d-41f6-b065-677b3e6c8523" containerName="revoke-keys"
Jul 17 19:32:31.686178 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.686048    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f371021-df80-4915-b8fe-13ebc1656423" containerName="revoke-keys"
Jul 17 19:32:31.686178 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.686053    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f371021-df80-4915-b8fe-13ebc1656423" containerName="revoke-keys"
Jul 17 19:32:31.686178 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.686111    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="105a5e13-7e1d-41f6-b065-677b3e6c8523" containerName="revoke-keys"
Jul 17 19:32:31.686178 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.686117    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="105a5e13-7e1d-41f6-b065-677b3e6c8523" containerName="revoke-keys"
Jul 17 19:32:31.686178 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.686124    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f371021-df80-4915-b8fe-13ebc1656423" containerName="revoke-keys"
Jul 17 19:32:31.686178 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.686130    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f371021-df80-4915-b8fe-13ebc1656423" containerName="revoke-keys"
Jul 17 19:32:31.686178 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.686137    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f371021-df80-4915-b8fe-13ebc1656423" containerName="revoke-keys"
Jul 17 19:32:31.686178 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.686144    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f0291f2-b2ce-470d-b219-bd23bfdc8e6d" containerName="revoke-keys"
Jul 17 19:32:31.689204 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.689187    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j"
Jul 17 19:32:31.691585 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.691566    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:32:31.696505 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.696482    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j"]
Jul 17 19:32:31.809140 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.809110    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64520fa-e0cb-40f3-b404-499dbb851eb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j\" (UID: \"d64520fa-e0cb-40f3-b404-499dbb851eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j"
Jul 17 19:32:31.910564 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.910534    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64520fa-e0cb-40f3-b404-499dbb851eb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j\" (UID: \"d64520fa-e0cb-40f3-b404-499dbb851eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j"
Jul 17 19:32:31.911122 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:31.911102    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64520fa-e0cb-40f3-b404-499dbb851eb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j\" (UID: \"d64520fa-e0cb-40f3-b404-499dbb851eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j"
Jul 17 19:32:32.001553 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:32.001496    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j"
Jul 17 19:32:32.134994 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:32.134969    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j"]
Jul 17 19:32:32.135966 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:32:32.135946    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64520fa_e0cb_40f3_b404_499dbb851eb8.slice/crio-0be87052d821f20bc4fb4855aa182805c6c4b804ec224bcd5da9ec610f09ee4d WatchSource:0}: Error finding container 0be87052d821f20bc4fb4855aa182805c6c4b804ec224bcd5da9ec610f09ee4d: Status 404 returned error can't find the container with id 0be87052d821f20bc4fb4855aa182805c6c4b804ec224bcd5da9ec610f09ee4d
Jul 17 19:32:32.233398 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:32.233363    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j" event={"ID":"d64520fa-e0cb-40f3-b404-499dbb851eb8","Type":"ContainerStarted","Data":"0be87052d821f20bc4fb4855aa182805c6c4b804ec224bcd5da9ec610f09ee4d"}
Jul 17 19:32:32.828234 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:32.828205    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7735-139962106418736"]
Jul 17 19:32:32.828608 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:32.828596    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f371021-df80-4915-b8fe-13ebc1656423" containerName="revoke-keys"
Jul 17 19:32:32.828649 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:32.828609    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f371021-df80-4915-b8fe-13ebc1656423" containerName="revoke-keys"
Jul 17 19:32:32.828649 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:32.828633    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="105a5e13-7e1d-41f6-b065-677b3e6c8523" containerName="revoke-keys"
Jul 17 19:32:32.828649 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:32.828638    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="105a5e13-7e1d-41f6-b065-677b3e6c8523" containerName="revoke-keys"
Jul 17 19:32:32.828755 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:32.828692    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="105a5e13-7e1d-41f6-b065-677b3e6c8523" containerName="revoke-keys"
Jul 17 19:32:32.831581 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:32.831565    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106418736"
Jul 17 19:32:32.833992 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:32.833974    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v6jdh\""
Jul 17 19:32:32.838353 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:32.838333    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-139962106418736"]
Jul 17 19:32:32.918649 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:32.918624    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbfnj\" (UniqueName: \"kubernetes.io/projected/a96d895b-645e-4520-86b8-e9299265a435-kube-api-access-sbfnj\") pod \"test-curl-7735-139962106418736\" (UID: \"a96d895b-645e-4520-86b8-e9299265a435\") " pod="opendatahub/test-curl-7735-139962106418736"
Jul 17 19:32:33.019724 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:33.019653    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbfnj\" (UniqueName: \"kubernetes.io/projected/a96d895b-645e-4520-86b8-e9299265a435-kube-api-access-sbfnj\") pod \"test-curl-7735-139962106418736\" (UID: \"a96d895b-645e-4520-86b8-e9299265a435\") " pod="opendatahub/test-curl-7735-139962106418736"
Jul 17 19:32:33.028678 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:33.028649    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbfnj\" (UniqueName: \"kubernetes.io/projected/a96d895b-645e-4520-86b8-e9299265a435-kube-api-access-sbfnj\") pod \"test-curl-7735-139962106418736\" (UID: \"a96d895b-645e-4520-86b8-e9299265a435\") " pod="opendatahub/test-curl-7735-139962106418736"
Jul 17 19:32:33.141919 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:33.141892    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106418736"
Jul 17 19:32:33.243944 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:33.243874    2559 generic.go:356] "Generic (PLEG): container finished" podID="d64520fa-e0cb-40f3-b404-499dbb851eb8" containerID="8ddcd2a44983537fcbf076987e1dfb3e63945e10988ca893108dd5a3fdcca91e" exitCode=6
Jul 17 19:32:33.244088 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:33.243968    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j" event={"ID":"d64520fa-e0cb-40f3-b404-499dbb851eb8","Type":"ContainerDied","Data":"8ddcd2a44983537fcbf076987e1dfb3e63945e10988ca893108dd5a3fdcca91e"}
Jul 17 19:32:33.244173 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:33.244116    2559 scope.go:117] "RemoveContainer" containerID="8ddcd2a44983537fcbf076987e1dfb3e63945e10988ca893108dd5a3fdcca91e"
Jul 17 19:32:33.277909 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:33.277884    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-139962106418736"]
Jul 17 19:32:33.279372 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:32:33.279347    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda96d895b_645e_4520_86b8_e9299265a435.slice/crio-989e132f160f58216d745412f9096d83d48b124509c365c9d4042b23e8d13c1c WatchSource:0}: Error finding container 989e132f160f58216d745412f9096d83d48b124509c365c9d4042b23e8d13c1c: Status 404 returned error can't find the container with id 989e132f160f58216d745412f9096d83d48b124509c365c9d4042b23e8d13c1c
Jul 17 19:32:34.251078 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:34.250999    2559 generic.go:356] "Generic (PLEG): container finished" podID="a96d895b-645e-4520-86b8-e9299265a435" containerID="e663e17c01988ffc5dff22bfb0e6355b0b1dbce4fafe3032b79e3da3d73a6068" exitCode=0
Jul 17 19:32:34.251489 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:34.251076    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-139962106418736" event={"ID":"a96d895b-645e-4520-86b8-e9299265a435","Type":"ContainerDied","Data":"e663e17c01988ffc5dff22bfb0e6355b0b1dbce4fafe3032b79e3da3d73a6068"}
Jul 17 19:32:34.251489 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:34.251117    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-139962106418736" event={"ID":"a96d895b-645e-4520-86b8-e9299265a435","Type":"ContainerStarted","Data":"989e132f160f58216d745412f9096d83d48b124509c365c9d4042b23e8d13c1c"}
Jul 17 19:32:34.253004 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:34.252985    2559 generic.go:356] "Generic (PLEG): container finished" podID="d64520fa-e0cb-40f3-b404-499dbb851eb8" containerID="3826dd829169797717a89d2fc9b37c590d3859729544aef94a81068e30fe1a3b" exitCode=6
Jul 17 19:32:34.253091 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:34.253024    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j" event={"ID":"d64520fa-e0cb-40f3-b404-499dbb851eb8","Type":"ContainerDied","Data":"3826dd829169797717a89d2fc9b37c590d3859729544aef94a81068e30fe1a3b"}
Jul 17 19:32:34.253091 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:34.253066    2559 scope.go:117] "RemoveContainer" containerID="8ddcd2a44983537fcbf076987e1dfb3e63945e10988ca893108dd5a3fdcca91e"
Jul 17 19:32:34.253265 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:34.253239    2559 scope.go:117] "RemoveContainer" containerID="3826dd829169797717a89d2fc9b37c590d3859729544aef94a81068e30fe1a3b"
Jul 17 19:32:34.253518 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:32:34.253495    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j_odh-ai-gateway-infra(d64520fa-e0cb-40f3-b404-499dbb851eb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j" podUID="d64520fa-e0cb-40f3-b404-499dbb851eb8"
Jul 17 19:32:35.260419 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:35.260395    2559 scope.go:117] "RemoveContainer" containerID="3826dd829169797717a89d2fc9b37c590d3859729544aef94a81068e30fe1a3b"
Jul 17 19:32:35.261078 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:32:35.260563    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j_odh-ai-gateway-infra(d64520fa-e0cb-40f3-b404-499dbb851eb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j" podUID="d64520fa-e0cb-40f3-b404-499dbb851eb8"
Jul 17 19:32:35.390035 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:35.390009    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106418736"
Jul 17 19:32:35.423990 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:35.423966    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7735-139962106418736_a96d895b-645e-4520-86b8-e9299265a435/test-curl-7735-139962106418736/0.log"
Jul 17 19:32:35.444171 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:35.444145    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbfnj\" (UniqueName: \"kubernetes.io/projected/a96d895b-645e-4520-86b8-e9299265a435-kube-api-access-sbfnj\") pod \"a96d895b-645e-4520-86b8-e9299265a435\" (UID: \"a96d895b-645e-4520-86b8-e9299265a435\") "
Jul 17 19:32:35.446223 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:35.446196    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7735-139962106418736"]
Jul 17 19:32:35.446566 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:35.446531    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96d895b-645e-4520-86b8-e9299265a435-kube-api-access-sbfnj" (OuterVolumeSpecName: "kube-api-access-sbfnj") pod "a96d895b-645e-4520-86b8-e9299265a435" (UID: "a96d895b-645e-4520-86b8-e9299265a435"). InnerVolumeSpecName "kube-api-access-sbfnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:32:35.448626 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:35.448609    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7735-139962106418736"]
Jul 17 19:32:35.546017 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:35.545956    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbfnj\" (UniqueName: \"kubernetes.io/projected/a96d895b-645e-4520-86b8-e9299265a435-kube-api-access-sbfnj\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:36.126119 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:36.126087    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a96d895b-645e-4520-86b8-e9299265a435" path="/var/lib/kubelet/pods/a96d895b-645e-4520-86b8-e9299265a435/volumes"
Jul 17 19:32:36.267668 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:36.267646    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106418736"
Jul 17 19:32:36.268158 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:36.267696    2559 scope.go:117] "RemoveContainer" containerID="e663e17c01988ffc5dff22bfb0e6355b0b1dbce4fafe3032b79e3da3d73a6068"
Jul 17 19:32:36.391235 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:36.391181    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7"]
Jul 17 19:32:36.391582 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:36.391569    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a96d895b-645e-4520-86b8-e9299265a435" containerName="test-curl-7735-139962106418736"
Jul 17 19:32:36.391633 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:36.391583    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a96d895b-645e-4520-86b8-e9299265a435" containerName="test-curl-7735-139962106418736"
Jul 17 19:32:36.391675 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:36.391665    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a96d895b-645e-4520-86b8-e9299265a435" containerName="test-curl-7735-139962106418736"
Jul 17 19:32:36.395722 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:36.395701    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7"
Jul 17 19:32:36.403458 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:36.403262    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7"]
Jul 17 19:32:36.453163 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:36.453138    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c66306b-9716-48f5-bca7-00dd1caed2cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7\" (UID: \"1c66306b-9716-48f5-bca7-00dd1caed2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7"
Jul 17 19:32:36.554411 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:36.554386    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c66306b-9716-48f5-bca7-00dd1caed2cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7\" (UID: \"1c66306b-9716-48f5-bca7-00dd1caed2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7"
Jul 17 19:32:36.554950 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:36.554931    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c66306b-9716-48f5-bca7-00dd1caed2cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7\" (UID: \"1c66306b-9716-48f5-bca7-00dd1caed2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7"
Jul 17 19:32:36.707177 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:36.707116    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7"
Jul 17 19:32:36.849174 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:36.849147    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7"]
Jul 17 19:32:36.850125 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:32:36.850099    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c66306b_9716_48f5_bca7_00dd1caed2cf.slice/crio-e22cdfe9999d8fbe8a7e1416e51801b9f45be750d38f89b2671fd9d523f69c7a WatchSource:0}: Error finding container e22cdfe9999d8fbe8a7e1416e51801b9f45be750d38f89b2671fd9d523f69c7a: Status 404 returned error can't find the container with id e22cdfe9999d8fbe8a7e1416e51801b9f45be750d38f89b2671fd9d523f69c7a
Jul 17 19:32:37.276738 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:37.276708    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7" event={"ID":"1c66306b-9716-48f5-bca7-00dd1caed2cf","Type":"ContainerStarted","Data":"927dd0bc1c630c4527a6bee69877bab48c523b1f9d1714dd41621d4a7aa4402b"}
Jul 17 19:32:37.277055 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:37.276764    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7" event={"ID":"1c66306b-9716-48f5-bca7-00dd1caed2cf","Type":"ContainerStarted","Data":"e22cdfe9999d8fbe8a7e1416e51801b9f45be750d38f89b2671fd9d523f69c7a"}
Jul 17 19:32:37.291942 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:37.291894    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7" podStartSLOduration=1.291880223 podStartE2EDuration="1.291880223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:32:37.290504625 +0000 UTC m=+3567.732013315" watchObservedRunningTime="2026-07-17 19:32:37.291880223 +0000 UTC m=+3567.733388911"
Jul 17 19:32:37.838434 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:37.838402    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7735-139962106361152"]
Jul 17 19:32:37.841948 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:37.841932    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106361152"
Jul 17 19:32:37.844989 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:37.844963    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v6jdh\""
Jul 17 19:32:37.848766 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:37.848580    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-139962106361152"]
Jul 17 19:32:37.968606 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:37.968576    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p7zp\" (UniqueName: \"kubernetes.io/projected/68517640-9fe9-4be3-8b83-e9de8e7730d0-kube-api-access-5p7zp\") pod \"test-curl-7735-139962106361152\" (UID: \"68517640-9fe9-4be3-8b83-e9de8e7730d0\") " pod="opendatahub/test-curl-7735-139962106361152"
Jul 17 19:32:38.069512 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:38.069485    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p7zp\" (UniqueName: \"kubernetes.io/projected/68517640-9fe9-4be3-8b83-e9de8e7730d0-kube-api-access-5p7zp\") pod \"test-curl-7735-139962106361152\" (UID: \"68517640-9fe9-4be3-8b83-e9de8e7730d0\") " pod="opendatahub/test-curl-7735-139962106361152"
Jul 17 19:32:38.077963 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:38.077936    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p7zp\" (UniqueName: \"kubernetes.io/projected/68517640-9fe9-4be3-8b83-e9de8e7730d0-kube-api-access-5p7zp\") pod \"test-curl-7735-139962106361152\" (UID: \"68517640-9fe9-4be3-8b83-e9de8e7730d0\") " pod="opendatahub/test-curl-7735-139962106361152"
Jul 17 19:32:38.152956 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:38.152903    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106361152"
Jul 17 19:32:38.285588 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:38.285557    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-139962106361152"]
Jul 17 19:32:38.285966 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:32:38.285928    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68517640_9fe9_4be3_8b83_e9de8e7730d0.slice/crio-d473fa52cc9a550df567d3b2fce83213a10018f57eadf0fe07625b01603de4a5 WatchSource:0}: Error finding container d473fa52cc9a550df567d3b2fce83213a10018f57eadf0fe07625b01603de4a5: Status 404 returned error can't find the container with id d473fa52cc9a550df567d3b2fce83213a10018f57eadf0fe07625b01603de4a5
Jul 17 19:32:38.287056 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:38.287022    2559 generic.go:356] "Generic (PLEG): container finished" podID="1c66306b-9716-48f5-bca7-00dd1caed2cf" containerID="927dd0bc1c630c4527a6bee69877bab48c523b1f9d1714dd41621d4a7aa4402b" exitCode=6
Jul 17 19:32:38.287170 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:38.287111    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7" event={"ID":"1c66306b-9716-48f5-bca7-00dd1caed2cf","Type":"ContainerDied","Data":"927dd0bc1c630c4527a6bee69877bab48c523b1f9d1714dd41621d4a7aa4402b"}
Jul 17 19:32:38.287381 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:38.287360    2559 scope.go:117] "RemoveContainer" containerID="927dd0bc1c630c4527a6bee69877bab48c523b1f9d1714dd41621d4a7aa4402b"
Jul 17 19:32:39.294716 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:39.294637    2559 generic.go:356] "Generic (PLEG): container finished" podID="68517640-9fe9-4be3-8b83-e9de8e7730d0" containerID="a98a0bb6cd83d6cc7b41759ecce9628f1bfbea3724444aedc739d9a4ae99a8b5" exitCode=0
Jul 17 19:32:39.295156 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:39.294725    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-139962106361152" event={"ID":"68517640-9fe9-4be3-8b83-e9de8e7730d0","Type":"ContainerDied","Data":"a98a0bb6cd83d6cc7b41759ecce9628f1bfbea3724444aedc739d9a4ae99a8b5"}
Jul 17 19:32:39.295156 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:39.294775    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-139962106361152" event={"ID":"68517640-9fe9-4be3-8b83-e9de8e7730d0","Type":"ContainerStarted","Data":"d473fa52cc9a550df567d3b2fce83213a10018f57eadf0fe07625b01603de4a5"}
Jul 17 19:32:39.296681 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:39.296656    2559 generic.go:356] "Generic (PLEG): container finished" podID="1c66306b-9716-48f5-bca7-00dd1caed2cf" containerID="62b27033a40c0ce17a98ceba120d1d1fcfbb27852bce23a3e3f2c6a838cc12e7" exitCode=6
Jul 17 19:32:39.296818 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:39.296704    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7" event={"ID":"1c66306b-9716-48f5-bca7-00dd1caed2cf","Type":"ContainerDied","Data":"62b27033a40c0ce17a98ceba120d1d1fcfbb27852bce23a3e3f2c6a838cc12e7"}
Jul 17 19:32:39.296818 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:39.296747    2559 scope.go:117] "RemoveContainer" containerID="927dd0bc1c630c4527a6bee69877bab48c523b1f9d1714dd41621d4a7aa4402b"
Jul 17 19:32:39.296942 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:39.296926    2559 scope.go:117] "RemoveContainer" containerID="62b27033a40c0ce17a98ceba120d1d1fcfbb27852bce23a3e3f2c6a838cc12e7"
Jul 17 19:32:39.297163 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:32:39.297142    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7_odh-ai-gateway-infra(1c66306b-9716-48f5-bca7-00dd1caed2cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7" podUID="1c66306b-9716-48f5-bca7-00dd1caed2cf"
Jul 17 19:32:40.304560 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:40.304528    2559 scope.go:117] "RemoveContainer" containerID="62b27033a40c0ce17a98ceba120d1d1fcfbb27852bce23a3e3f2c6a838cc12e7"
Jul 17 19:32:40.304941 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:32:40.304792    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7_odh-ai-gateway-infra(1c66306b-9716-48f5-bca7-00dd1caed2cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7" podUID="1c66306b-9716-48f5-bca7-00dd1caed2cf"
Jul 17 19:32:40.444366 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:40.444339    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106361152"
Jul 17 19:32:40.464653 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:40.464632    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7735-139962106361152_68517640-9fe9-4be3-8b83-e9de8e7730d0/test-curl-7735-139962106361152/0.log"
Jul 17 19:32:40.489461 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:40.489431    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7735-139962106361152"]
Jul 17 19:32:40.493846 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:40.493825    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7735-139962106361152"]
Jul 17 19:32:40.593038 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:40.593003    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p7zp\" (UniqueName: \"kubernetes.io/projected/68517640-9fe9-4be3-8b83-e9de8e7730d0-kube-api-access-5p7zp\") pod \"68517640-9fe9-4be3-8b83-e9de8e7730d0\" (UID: \"68517640-9fe9-4be3-8b83-e9de8e7730d0\") "
Jul 17 19:32:40.595390 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:40.595357    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68517640-9fe9-4be3-8b83-e9de8e7730d0-kube-api-access-5p7zp" (OuterVolumeSpecName: "kube-api-access-5p7zp") pod "68517640-9fe9-4be3-8b83-e9de8e7730d0" (UID: "68517640-9fe9-4be3-8b83-e9de8e7730d0"). InnerVolumeSpecName "kube-api-access-5p7zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:32:40.694417 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:40.694380    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p7zp\" (UniqueName: \"kubernetes.io/projected/68517640-9fe9-4be3-8b83-e9de8e7730d0-kube-api-access-5p7zp\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:41.312069 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:41.312043    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106361152"
Jul 17 19:32:41.312069 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:41.312064    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d473fa52cc9a550df567d3b2fce83213a10018f57eadf0fe07625b01603de4a5"
Jul 17 19:32:41.322047 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:32:41.322020    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-139962106361152\" is forbidden: User \"system:node:ip-10-0-138-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-51.ec2.internal' and this object" podUID="68517640-9fe9-4be3-8b83-e9de8e7730d0" pod="opendatahub/test-curl-7735-139962106361152"
Jul 17 19:32:42.127108 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:42.127066    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68517640-9fe9-4be3-8b83-e9de8e7730d0" path="/var/lib/kubelet/pods/68517640-9fe9-4be3-8b83-e9de8e7730d0/volumes"
Jul 17 19:32:42.873647 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:42.873617    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7735-139962106362160"]
Jul 17 19:32:42.874038 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:42.874027    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68517640-9fe9-4be3-8b83-e9de8e7730d0" containerName="test-curl-7735-139962106361152"
Jul 17 19:32:42.874038 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:42.874039    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="68517640-9fe9-4be3-8b83-e9de8e7730d0" containerName="test-curl-7735-139962106361152"
Jul 17 19:32:42.874130 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:42.874119    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="68517640-9fe9-4be3-8b83-e9de8e7730d0" containerName="test-curl-7735-139962106361152"
Jul 17 19:32:42.877458 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:42.877441    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106362160"
Jul 17 19:32:42.879883 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:42.879863    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v6jdh\""
Jul 17 19:32:42.885114 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:42.884791    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-139962106362160"]
Jul 17 19:32:43.016258 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.016220    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmcmz\" (UniqueName: \"kubernetes.io/projected/6ad2d715-2ad4-4fbf-9c2b-1632bfc22251-kube-api-access-kmcmz\") pod \"test-curl-7735-139962106362160\" (UID: \"6ad2d715-2ad4-4fbf-9c2b-1632bfc22251\") " pod="opendatahub/test-curl-7735-139962106362160"
Jul 17 19:32:43.117071 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.117019    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmcmz\" (UniqueName: \"kubernetes.io/projected/6ad2d715-2ad4-4fbf-9c2b-1632bfc22251-kube-api-access-kmcmz\") pod \"test-curl-7735-139962106362160\" (UID: \"6ad2d715-2ad4-4fbf-9c2b-1632bfc22251\") " pod="opendatahub/test-curl-7735-139962106362160"
Jul 17 19:32:43.125400 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.125338    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmcmz\" (UniqueName: \"kubernetes.io/projected/6ad2d715-2ad4-4fbf-9c2b-1632bfc22251-kube-api-access-kmcmz\") pod \"test-curl-7735-139962106362160\" (UID: \"6ad2d715-2ad4-4fbf-9c2b-1632bfc22251\") " pod="opendatahub/test-curl-7735-139962106362160"
Jul 17 19:32:43.155423 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.155382    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk" podUID="7d737418-1fcb-439e-be0b-35ec5020616e" containerName="main" containerID="cri-o://c479c4d0023a1b37155fc5ef0c1bb466691d544ffc62c06ecf54027ff869937e" gracePeriod=45
Jul 17 19:32:43.187838 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.187804    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106362160"
Jul 17 19:32:43.327966 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.327909    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-139962106362160"]
Jul 17 19:32:43.329944 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:32:43.329913    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ad2d715_2ad4_4fbf_9c2b_1632bfc22251.slice/crio-0cb821c0031c9cdf9dd207fedbaf552c7a3b78bacb3e07d12a2e55a1fd9efabf WatchSource:0}: Error finding container 0cb821c0031c9cdf9dd207fedbaf552c7a3b78bacb3e07d12a2e55a1fd9efabf: Status 404 returned error can't find the container with id 0cb821c0031c9cdf9dd207fedbaf552c7a3b78bacb3e07d12a2e55a1fd9efabf
Jul 17 19:32:43.335388 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.335362    2559 generic.go:356] "Generic (PLEG): container finished" podID="7d737418-1fcb-439e-be0b-35ec5020616e" containerID="c479c4d0023a1b37155fc5ef0c1bb466691d544ffc62c06ecf54027ff869937e" exitCode=0
Jul 17 19:32:43.335484 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.335429    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk" event={"ID":"7d737418-1fcb-439e-be0b-35ec5020616e","Type":"ContainerDied","Data":"c479c4d0023a1b37155fc5ef0c1bb466691d544ffc62c06ecf54027ff869937e"}
Jul 17 19:32:43.420458 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.420434    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:32:43.521431 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.521399    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q6sv\" (UniqueName: \"kubernetes.io/projected/7d737418-1fcb-439e-be0b-35ec5020616e-kube-api-access-5q6sv\") pod \"7d737418-1fcb-439e-be0b-35ec5020616e\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") "
Jul 17 19:32:43.521625 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.521448    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-home\") pod \"7d737418-1fcb-439e-be0b-35ec5020616e\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") "
Jul 17 19:32:43.521625 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.521474    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-model-cache\") pod \"7d737418-1fcb-439e-be0b-35ec5020616e\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") "
Jul 17 19:32:43.521625 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.521582    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-dshm\") pod \"7d737418-1fcb-439e-be0b-35ec5020616e\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") "
Jul 17 19:32:43.521625 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.521622    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d737418-1fcb-439e-be0b-35ec5020616e-tls-certs\") pod \"7d737418-1fcb-439e-be0b-35ec5020616e\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") "
Jul 17 19:32:43.521835 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.521655    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-tmp-dir\") pod \"7d737418-1fcb-439e-be0b-35ec5020616e\" (UID: \"7d737418-1fcb-439e-be0b-35ec5020616e\") "
Jul 17 19:32:43.521835 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.521690    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d737418-1fcb-439e-be0b-35ec5020616e" (UID: "7d737418-1fcb-439e-be0b-35ec5020616e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:32:43.521835 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.521708    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-home" (OuterVolumeSpecName: "home") pod "7d737418-1fcb-439e-be0b-35ec5020616e" (UID: "7d737418-1fcb-439e-be0b-35ec5020616e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:32:43.521971 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.521926    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d737418-1fcb-439e-be0b-35ec5020616e" (UID: "7d737418-1fcb-439e-be0b-35ec5020616e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:32:43.522052 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.522036    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-home\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:43.522094 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.522056    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-model-cache\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:43.522094 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.522066    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-tmp-dir\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:43.523775 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.523751    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d737418-1fcb-439e-be0b-35ec5020616e-kube-api-access-5q6sv" (OuterVolumeSpecName: "kube-api-access-5q6sv") pod "7d737418-1fcb-439e-be0b-35ec5020616e" (UID: "7d737418-1fcb-439e-be0b-35ec5020616e"). InnerVolumeSpecName "kube-api-access-5q6sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:32:43.524230 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.524215    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d737418-1fcb-439e-be0b-35ec5020616e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d737418-1fcb-439e-be0b-35ec5020616e" (UID: "7d737418-1fcb-439e-be0b-35ec5020616e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:32:43.524230 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.524220    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-dshm" (OuterVolumeSpecName: "dshm") pod "7d737418-1fcb-439e-be0b-35ec5020616e" (UID: "7d737418-1fcb-439e-be0b-35ec5020616e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:32:43.623103 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.623077    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q6sv\" (UniqueName: \"kubernetes.io/projected/7d737418-1fcb-439e-be0b-35ec5020616e-kube-api-access-5q6sv\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:43.623218 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.623106    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d737418-1fcb-439e-be0b-35ec5020616e-dshm\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:43.623218 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.623121    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d737418-1fcb-439e-be0b-35ec5020616e-tls-certs\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:43.952081 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:43.951962    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w" podUID="f3eb86c7-e264-4da9-9992-d80b9d3051f7" containerName="main" containerID="cri-o://c636de237090c6f13130b993fc4a6b688922b361692e4956d88c0e6c143f8448" gracePeriod=45
Jul 17 19:32:44.223397 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.223373    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:32:44.328404 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.328369    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-home\") pod \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") "
Jul 17 19:32:44.328576 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.328433    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-dshm\") pod \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") "
Jul 17 19:32:44.328576 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.328466    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkx2n\" (UniqueName: \"kubernetes.io/projected/f3eb86c7-e264-4da9-9992-d80b9d3051f7-kube-api-access-lkx2n\") pod \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") "
Jul 17 19:32:44.328576 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.328497    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-model-cache\") pod \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") "
Jul 17 19:32:44.328765 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.328641    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3eb86c7-e264-4da9-9992-d80b9d3051f7-tls-certs\") pod \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") "
Jul 17 19:32:44.328765 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.328679    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-tmp-dir\") pod \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\" (UID: \"f3eb86c7-e264-4da9-9992-d80b9d3051f7\") "
Jul 17 19:32:44.328765 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.328720    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-home" (OuterVolumeSpecName: "home") pod "f3eb86c7-e264-4da9-9992-d80b9d3051f7" (UID: "f3eb86c7-e264-4da9-9992-d80b9d3051f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:32:44.328929 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.328748    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3eb86c7-e264-4da9-9992-d80b9d3051f7" (UID: "f3eb86c7-e264-4da9-9992-d80b9d3051f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:32:44.328986 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.328935    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3eb86c7-e264-4da9-9992-d80b9d3051f7" (UID: "f3eb86c7-e264-4da9-9992-d80b9d3051f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:32:44.329076 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.329060    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-home\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:44.329211 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.329079    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-model-cache\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:44.329211 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.329090    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-tmp-dir\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:44.330755 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.330730    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-dshm" (OuterVolumeSpecName: "dshm") pod "f3eb86c7-e264-4da9-9992-d80b9d3051f7" (UID: "f3eb86c7-e264-4da9-9992-d80b9d3051f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:32:44.330856 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.330817    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3eb86c7-e264-4da9-9992-d80b9d3051f7-kube-api-access-lkx2n" (OuterVolumeSpecName: "kube-api-access-lkx2n") pod "f3eb86c7-e264-4da9-9992-d80b9d3051f7" (UID: "f3eb86c7-e264-4da9-9992-d80b9d3051f7"). InnerVolumeSpecName "kube-api-access-lkx2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:32:44.330856 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.330834    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3eb86c7-e264-4da9-9992-d80b9d3051f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3eb86c7-e264-4da9-9992-d80b9d3051f7" (UID: "f3eb86c7-e264-4da9-9992-d80b9d3051f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:32:44.344031 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.344005    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk" event={"ID":"7d737418-1fcb-439e-be0b-35ec5020616e","Type":"ContainerDied","Data":"34bededdd460703bef0f0ee931b9d122f1fda9dbe04876e42f5b6d4c7f397590"}
Jul 17 19:32:44.344143 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.344061    2559 scope.go:117] "RemoveContainer" containerID="c479c4d0023a1b37155fc5ef0c1bb466691d544ffc62c06ecf54027ff869937e"
Jul 17 19:32:44.344143 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.344012    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"
Jul 17 19:32:44.345497 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.345474    2559 generic.go:356] "Generic (PLEG): container finished" podID="f3eb86c7-e264-4da9-9992-d80b9d3051f7" containerID="c636de237090c6f13130b993fc4a6b688922b361692e4956d88c0e6c143f8448" exitCode=0
Jul 17 19:32:44.345611 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.345515    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"
Jul 17 19:32:44.345611 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.345563    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w" event={"ID":"f3eb86c7-e264-4da9-9992-d80b9d3051f7","Type":"ContainerDied","Data":"c636de237090c6f13130b993fc4a6b688922b361692e4956d88c0e6c143f8448"}
Jul 17 19:32:44.345611 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.345604    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w" event={"ID":"f3eb86c7-e264-4da9-9992-d80b9d3051f7","Type":"ContainerDied","Data":"f1e0558f368f05d39c546b52412e1a98069dff07bc737ce984379d0a27f50471"}
Jul 17 19:32:44.347680 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.347654    2559 generic.go:356] "Generic (PLEG): container finished" podID="6ad2d715-2ad4-4fbf-9c2b-1632bfc22251" containerID="9f251e0be6ef48085ff9e834764fe3f1cbb5585a46012b195b5f6108aa02aaf2" exitCode=0
Jul 17 19:32:44.347797 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.347744    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-139962106362160" event={"ID":"6ad2d715-2ad4-4fbf-9c2b-1632bfc22251","Type":"ContainerDied","Data":"9f251e0be6ef48085ff9e834764fe3f1cbb5585a46012b195b5f6108aa02aaf2"}
Jul 17 19:32:44.347797 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.347776    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-139962106362160" event={"ID":"6ad2d715-2ad4-4fbf-9c2b-1632bfc22251","Type":"ContainerStarted","Data":"0cb821c0031c9cdf9dd207fedbaf552c7a3b78bacb3e07d12a2e55a1fd9efabf"}
Jul 17 19:32:44.354084 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.354069    2559 scope.go:117] "RemoveContainer" containerID="c636de237090c6f13130b993fc4a6b688922b361692e4956d88c0e6c143f8448"
Jul 17 19:32:44.362427 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.362398    2559 scope.go:117] "RemoveContainer" containerID="c636de237090c6f13130b993fc4a6b688922b361692e4956d88c0e6c143f8448"
Jul 17 19:32:44.362796 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:32:44.362772    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c636de237090c6f13130b993fc4a6b688922b361692e4956d88c0e6c143f8448\": container with ID starting with c636de237090c6f13130b993fc4a6b688922b361692e4956d88c0e6c143f8448 not found: ID does not exist" containerID="c636de237090c6f13130b993fc4a6b688922b361692e4956d88c0e6c143f8448"
Jul 17 19:32:44.362920 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.362900    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c636de237090c6f13130b993fc4a6b688922b361692e4956d88c0e6c143f8448"} err="failed to get container status \"c636de237090c6f13130b993fc4a6b688922b361692e4956d88c0e6c143f8448\": rpc error: code = NotFound desc = could not find container \"c636de237090c6f13130b993fc4a6b688922b361692e4956d88c0e6c143f8448\": container with ID starting with c636de237090c6f13130b993fc4a6b688922b361692e4956d88c0e6c143f8448 not found: ID does not exist"
Jul 17 19:32:44.379065 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.379046    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"]
Jul 17 19:32:44.382679 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.382661    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-98e305/rate-test-model-98e305-kserve-85b7cd595f-hppzk"]
Jul 17 19:32:44.393139 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.393119    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"]
Jul 17 19:32:44.397153 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.397135    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-2109fd/rate-test-model-2109fd-kserve-6df5445575-9gt4w"]
Jul 17 19:32:44.429813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.429780    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3eb86c7-e264-4da9-9992-d80b9d3051f7-tls-certs\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:44.429813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.429812    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3eb86c7-e264-4da9-9992-d80b9d3051f7-dshm\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:44.429813 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.429822    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkx2n\" (UniqueName: \"kubernetes.io/projected/f3eb86c7-e264-4da9-9992-d80b9d3051f7-kube-api-access-lkx2n\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:44.580700 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.580665    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7"]
Jul 17 19:32:44.581103 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.581090    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d737418-1fcb-439e-be0b-35ec5020616e" containerName="main"
Jul 17 19:32:44.581163 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.581106    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d737418-1fcb-439e-be0b-35ec5020616e" containerName="main"
Jul 17 19:32:44.581163 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.581115    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3eb86c7-e264-4da9-9992-d80b9d3051f7" containerName="main"
Jul 17 19:32:44.581163 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.581121    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3eb86c7-e264-4da9-9992-d80b9d3051f7" containerName="main"
Jul 17 19:32:44.581278 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.581206    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d737418-1fcb-439e-be0b-35ec5020616e" containerName="main"
Jul 17 19:32:44.581278 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.581217    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3eb86c7-e264-4da9-9992-d80b9d3051f7" containerName="main"
Jul 17 19:32:44.585690 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.585673    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7"
Jul 17 19:32:44.591545 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.591519    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7"]
Jul 17 19:32:44.733276 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.733244    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac95972f-340f-40b2-b781-3e8a18f9bf9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7\" (UID: \"ac95972f-340f-40b2-b781-3e8a18f9bf9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7"
Jul 17 19:32:44.834640 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.834560    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac95972f-340f-40b2-b781-3e8a18f9bf9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7\" (UID: \"ac95972f-340f-40b2-b781-3e8a18f9bf9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7"
Jul 17 19:32:44.835139 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.835121    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac95972f-340f-40b2-b781-3e8a18f9bf9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7\" (UID: \"ac95972f-340f-40b2-b781-3e8a18f9bf9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7"
Jul 17 19:32:44.897701 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:44.897672    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7"
Jul 17 19:32:45.034812 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:45.034786    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7"]
Jul 17 19:32:45.036698 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:32:45.036672    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac95972f_340f_40b2_b781_3e8a18f9bf9e.slice/crio-5e9dcc64af6863ece91166d7bb18c5b4925e085e40204b7c1a622c5169bbe449 WatchSource:0}: Error finding container 5e9dcc64af6863ece91166d7bb18c5b4925e085e40204b7c1a622c5169bbe449: Status 404 returned error can't find the container with id 5e9dcc64af6863ece91166d7bb18c5b4925e085e40204b7c1a622c5169bbe449
Jul 17 19:32:45.359854 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:45.359825    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7" event={"ID":"ac95972f-340f-40b2-b781-3e8a18f9bf9e","Type":"ContainerStarted","Data":"5e9dcc64af6863ece91166d7bb18c5b4925e085e40204b7c1a622c5169bbe449"}
Jul 17 19:32:45.498668 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:45.498634    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106362160"
Jul 17 19:32:45.518148 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:45.518123    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7735-139962106362160_6ad2d715-2ad4-4fbf-9c2b-1632bfc22251/test-curl-7735-139962106362160/0.log"
Jul 17 19:32:45.544199 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:45.544170    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7735-139962106362160"]
Jul 17 19:32:45.546516 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:45.546496    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7735-139962106362160"]
Jul 17 19:32:45.643138 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:45.643046    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmcmz\" (UniqueName: \"kubernetes.io/projected/6ad2d715-2ad4-4fbf-9c2b-1632bfc22251-kube-api-access-kmcmz\") pod \"6ad2d715-2ad4-4fbf-9c2b-1632bfc22251\" (UID: \"6ad2d715-2ad4-4fbf-9c2b-1632bfc22251\") "
Jul 17 19:32:45.645327 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:45.645287    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad2d715-2ad4-4fbf-9c2b-1632bfc22251-kube-api-access-kmcmz" (OuterVolumeSpecName: "kube-api-access-kmcmz") pod "6ad2d715-2ad4-4fbf-9c2b-1632bfc22251" (UID: "6ad2d715-2ad4-4fbf-9c2b-1632bfc22251"). InnerVolumeSpecName "kube-api-access-kmcmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:32:45.722837 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:45.722805    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-98e305-zkxlm"]
Jul 17 19:32:45.723228 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:45.723214    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad2d715-2ad4-4fbf-9c2b-1632bfc22251" containerName="test-curl-7735-139962106362160"
Jul 17 19:32:45.723278 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:45.723230    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad2d715-2ad4-4fbf-9c2b-1632bfc22251" containerName="test-curl-7735-139962106362160"
Jul 17 19:32:45.723332 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:45.723326    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad2d715-2ad4-4fbf-9c2b-1632bfc22251" containerName="test-curl-7735-139962106362160"
Jul 17 19:32:45.726435 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:45.726420    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-98e305-zkxlm"
Jul 17 19:32:45.734545 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:45.734269    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-98e305-zkxlm"]
Jul 17 19:32:45.744694 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:45.744669    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmcmz\" (UniqueName: \"kubernetes.io/projected/6ad2d715-2ad4-4fbf-9c2b-1632bfc22251-kube-api-access-kmcmz\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:45.845768 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:45.845727    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fed73b5-5866-44c5-b044-b2661f39fce3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-98e305-zkxlm\" (UID: \"6fed73b5-5866-44c5-b044-b2661f39fce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-98e305-zkxlm"
Jul 17 19:32:45.946636 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:45.946610    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fed73b5-5866-44c5-b044-b2661f39fce3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-98e305-zkxlm\" (UID: \"6fed73b5-5866-44c5-b044-b2661f39fce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-98e305-zkxlm"
Jul 17 19:32:45.947244 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:45.947227    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fed73b5-5866-44c5-b044-b2661f39fce3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-98e305-zkxlm\" (UID: \"6fed73b5-5866-44c5-b044-b2661f39fce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-98e305-zkxlm"
Jul 17 19:32:46.040569 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:46.040537    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-98e305-zkxlm"
Jul 17 19:32:46.137802 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:46.134842    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ad2d715-2ad4-4fbf-9c2b-1632bfc22251" path="/var/lib/kubelet/pods/6ad2d715-2ad4-4fbf-9c2b-1632bfc22251/volumes"
Jul 17 19:32:46.137802 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:46.135503    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d737418-1fcb-439e-be0b-35ec5020616e" path="/var/lib/kubelet/pods/7d737418-1fcb-439e-be0b-35ec5020616e/volumes"
Jul 17 19:32:46.137802 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:46.136131    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3eb86c7-e264-4da9-9992-d80b9d3051f7" path="/var/lib/kubelet/pods/f3eb86c7-e264-4da9-9992-d80b9d3051f7/volumes"
Jul 17 19:32:46.193809 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:46.193783    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-98e305-zkxlm"]
Jul 17 19:32:46.194484 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:32:46.194459    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fed73b5_5866_44c5_b044_b2661f39fce3.slice/crio-4d665e3a5ade00f4e474d7437b802765bba088cd308623e4f9dd8ad3af9383e6 WatchSource:0}: Error finding container 4d665e3a5ade00f4e474d7437b802765bba088cd308623e4f9dd8ad3af9383e6: Status 404 returned error can't find the container with id 4d665e3a5ade00f4e474d7437b802765bba088cd308623e4f9dd8ad3af9383e6
Jul 17 19:32:46.367677 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:46.367643    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-98e305-zkxlm" event={"ID":"6fed73b5-5866-44c5-b044-b2661f39fce3","Type":"ContainerStarted","Data":"4d665e3a5ade00f4e474d7437b802765bba088cd308623e4f9dd8ad3af9383e6"}
Jul 17 19:32:46.368957 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:46.368933    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106362160"
Jul 17 19:32:46.369108 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:46.368957    2559 scope.go:117] "RemoveContainer" containerID="9f251e0be6ef48085ff9e834764fe3f1cbb5585a46012b195b5f6108aa02aaf2"
Jul 17 19:32:46.370778 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:46.370756    2559 generic.go:356] "Generic (PLEG): container finished" podID="ac95972f-340f-40b2-b781-3e8a18f9bf9e" containerID="c08dd62516c4536f70d577883b87f2608a0eab4ac6d3c892d76f7da1ba759db1" exitCode=6
Jul 17 19:32:46.370873 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:46.370825    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7" event={"ID":"ac95972f-340f-40b2-b781-3e8a18f9bf9e","Type":"ContainerDied","Data":"c08dd62516c4536f70d577883b87f2608a0eab4ac6d3c892d76f7da1ba759db1"}
Jul 17 19:32:46.371057 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:46.371039    2559 scope.go:117] "RemoveContainer" containerID="c08dd62516c4536f70d577883b87f2608a0eab4ac6d3c892d76f7da1ba759db1"
Jul 17 19:32:47.378288 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:47.378255    2559 generic.go:356] "Generic (PLEG): container finished" podID="6fed73b5-5866-44c5-b044-b2661f39fce3" containerID="1d34cc49bd1577a0c952be5c7f46da47342221d68c02f55f73f8165d6ac33931" exitCode=0
Jul 17 19:32:47.378730 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:47.378342    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-98e305-zkxlm" event={"ID":"6fed73b5-5866-44c5-b044-b2661f39fce3","Type":"ContainerDied","Data":"1d34cc49bd1577a0c952be5c7f46da47342221d68c02f55f73f8165d6ac33931"}
Jul 17 19:32:47.380955 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:47.380926    2559 generic.go:356] "Generic (PLEG): container finished" podID="ac95972f-340f-40b2-b781-3e8a18f9bf9e" containerID="68f2d77498034f9c9338be503f390db28e752ff0dcb198029c5599ccfebae0a0" exitCode=6
Jul 17 19:32:47.381081 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:47.381002    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7" event={"ID":"ac95972f-340f-40b2-b781-3e8a18f9bf9e","Type":"ContainerDied","Data":"68f2d77498034f9c9338be503f390db28e752ff0dcb198029c5599ccfebae0a0"}
Jul 17 19:32:47.381081 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:47.381056    2559 scope.go:117] "RemoveContainer" containerID="c08dd62516c4536f70d577883b87f2608a0eab4ac6d3c892d76f7da1ba759db1"
Jul 17 19:32:47.381370 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:47.381355    2559 scope.go:117] "RemoveContainer" containerID="68f2d77498034f9c9338be503f390db28e752ff0dcb198029c5599ccfebae0a0"
Jul 17 19:32:47.381617 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:32:47.381583    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7_odh-ai-gateway-infra(ac95972f-340f-40b2-b781-3e8a18f9bf9e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7" podUID="ac95972f-340f-40b2-b781-3e8a18f9bf9e"
Jul 17 19:32:48.388964 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:48.388937    2559 scope.go:117] "RemoveContainer" containerID="68f2d77498034f9c9338be503f390db28e752ff0dcb198029c5599ccfebae0a0"
Jul 17 19:32:48.389406 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:32:48.389211    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7_odh-ai-gateway-infra(ac95972f-340f-40b2-b781-3e8a18f9bf9e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7" podUID="ac95972f-340f-40b2-b781-3e8a18f9bf9e"
Jul 17 19:32:48.520417 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:48.520394    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-98e305-zkxlm"
Jul 17 19:32:48.672061 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:48.671976    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fed73b5-5866-44c5-b044-b2661f39fce3-maas-api-service-ca\") pod \"6fed73b5-5866-44c5-b044-b2661f39fce3\" (UID: \"6fed73b5-5866-44c5-b044-b2661f39fce3\") "
Jul 17 19:32:48.672351 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:48.672330    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fed73b5-5866-44c5-b044-b2661f39fce3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6fed73b5-5866-44c5-b044-b2661f39fce3" (UID: "6fed73b5-5866-44c5-b044-b2661f39fce3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:32:48.773141 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:48.773113    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fed73b5-5866-44c5-b044-b2661f39fce3-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:49.395870 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:49.395841    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-98e305-zkxlm"
Jul 17 19:32:49.395870 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:49.395853    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-98e305-zkxlm" event={"ID":"6fed73b5-5866-44c5-b044-b2661f39fce3","Type":"ContainerDied","Data":"4d665e3a5ade00f4e474d7437b802765bba088cd308623e4f9dd8ad3af9383e6"}
Jul 17 19:32:49.396295 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:49.395903    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d665e3a5ade00f4e474d7437b802765bba088cd308623e4f9dd8ad3af9383e6"
Jul 17 19:32:50.123247 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:50.123216    2559 scope.go:117] "RemoveContainer" containerID="3826dd829169797717a89d2fc9b37c590d3859729544aef94a81068e30fe1a3b"
Jul 17 19:32:50.403762 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:50.403690    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j" event={"ID":"d64520fa-e0cb-40f3-b404-499dbb851eb8","Type":"ContainerStarted","Data":"4e624bfa39a4b4c82b203b599e91245ea299e10f303ffc842892601af543deda"}
Jul 17 19:32:50.419107 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:50.419060    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j" podStartSLOduration=19.419043882 podStartE2EDuration="19.419043882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:32:50.418475214 +0000 UTC m=+3580.859983905" watchObservedRunningTime="2026-07-17 19:32:50.419043882 +0000 UTC m=+3580.860552615"
Jul 17 19:32:51.120253 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:51.120225    2559 scope.go:117] "RemoveContainer" containerID="62b27033a40c0ce17a98ceba120d1d1fcfbb27852bce23a3e3f2c6a838cc12e7"
Jul 17 19:32:51.411019 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:51.410950    2559 generic.go:356] "Generic (PLEG): container finished" podID="d64520fa-e0cb-40f3-b404-499dbb851eb8" containerID="4e624bfa39a4b4c82b203b599e91245ea299e10f303ffc842892601af543deda" exitCode=6
Jul 17 19:32:51.411420 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:51.411026    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j" event={"ID":"d64520fa-e0cb-40f3-b404-499dbb851eb8","Type":"ContainerDied","Data":"4e624bfa39a4b4c82b203b599e91245ea299e10f303ffc842892601af543deda"}
Jul 17 19:32:51.411420 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:51.411085    2559 scope.go:117] "RemoveContainer" containerID="3826dd829169797717a89d2fc9b37c590d3859729544aef94a81068e30fe1a3b"
Jul 17 19:32:51.411420 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:51.411375    2559 scope.go:117] "RemoveContainer" containerID="4e624bfa39a4b4c82b203b599e91245ea299e10f303ffc842892601af543deda"
Jul 17 19:32:51.411623 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:32:51.411600    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j_odh-ai-gateway-infra(d64520fa-e0cb-40f3-b404-499dbb851eb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j" podUID="d64520fa-e0cb-40f3-b404-499dbb851eb8"
Jul 17 19:32:51.413138 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:51.413117    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7" event={"ID":"1c66306b-9716-48f5-bca7-00dd1caed2cf","Type":"ContainerStarted","Data":"3da5c215f668884255e0cbbb97e53b1cf75579ae30a7b7c38237a59641674de9"}
Jul 17 19:32:51.430697 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:51.430668    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j"]
Jul 17 19:32:52.423322 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:52.423263    2559 generic.go:356] "Generic (PLEG): container finished" podID="1c66306b-9716-48f5-bca7-00dd1caed2cf" containerID="3da5c215f668884255e0cbbb97e53b1cf75579ae30a7b7c38237a59641674de9" exitCode=6
Jul 17 19:32:52.423723 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:52.423333    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7" event={"ID":"1c66306b-9716-48f5-bca7-00dd1caed2cf","Type":"ContainerDied","Data":"3da5c215f668884255e0cbbb97e53b1cf75579ae30a7b7c38237a59641674de9"}
Jul 17 19:32:52.423723 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:52.423391    2559 scope.go:117] "RemoveContainer" containerID="62b27033a40c0ce17a98ceba120d1d1fcfbb27852bce23a3e3f2c6a838cc12e7"
Jul 17 19:32:52.423723 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:52.423660    2559 scope.go:117] "RemoveContainer" containerID="3da5c215f668884255e0cbbb97e53b1cf75579ae30a7b7c38237a59641674de9"
Jul 17 19:32:52.423935 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:32:52.423911    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7_odh-ai-gateway-infra(1c66306b-9716-48f5-bca7-00dd1caed2cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7" podUID="1c66306b-9716-48f5-bca7-00dd1caed2cf"
Jul 17 19:32:52.463946 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:52.463915    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7"]
Jul 17 19:32:52.560823 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:52.560795    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j"
Jul 17 19:32:52.710664 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:52.710578    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64520fa-e0cb-40f3-b404-499dbb851eb8-maas-api-service-ca\") pod \"d64520fa-e0cb-40f3-b404-499dbb851eb8\" (UID: \"d64520fa-e0cb-40f3-b404-499dbb851eb8\") "
Jul 17 19:32:52.710917 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:52.710896    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d64520fa-e0cb-40f3-b404-499dbb851eb8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d64520fa-e0cb-40f3-b404-499dbb851eb8" (UID: "d64520fa-e0cb-40f3-b404-499dbb851eb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:32:52.811377 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:52.811343    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64520fa-e0cb-40f3-b404-499dbb851eb8-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:53.431113 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.431082    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j"
Jul 17 19:32:53.431626 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.431124    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j" event={"ID":"d64520fa-e0cb-40f3-b404-499dbb851eb8","Type":"ContainerDied","Data":"0be87052d821f20bc4fb4855aa182805c6c4b804ec224bcd5da9ec610f09ee4d"}
Jul 17 19:32:53.431626 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.431189    2559 scope.go:117] "RemoveContainer" containerID="4e624bfa39a4b4c82b203b599e91245ea299e10f303ffc842892601af543deda"
Jul 17 19:32:53.460093 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.460066    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh"]
Jul 17 19:32:53.460509 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.460495    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d64520fa-e0cb-40f3-b404-499dbb851eb8" containerName="revoke-keys"
Jul 17 19:32:53.460569 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.460511    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d64520fa-e0cb-40f3-b404-499dbb851eb8" containerName="revoke-keys"
Jul 17 19:32:53.460569 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.460522    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d64520fa-e0cb-40f3-b404-499dbb851eb8" containerName="revoke-keys"
Jul 17 19:32:53.460569 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.460527    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d64520fa-e0cb-40f3-b404-499dbb851eb8" containerName="revoke-keys"
Jul 17 19:32:53.460569 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.460545    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d64520fa-e0cb-40f3-b404-499dbb851eb8" containerName="revoke-keys"
Jul 17 19:32:53.460569 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.460551    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d64520fa-e0cb-40f3-b404-499dbb851eb8" containerName="revoke-keys"
Jul 17 19:32:53.460569 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.460558    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fed73b5-5866-44c5-b044-b2661f39fce3" containerName="revoke-keys"
Jul 17 19:32:53.460569 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.460563    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fed73b5-5866-44c5-b044-b2661f39fce3" containerName="revoke-keys"
Jul 17 19:32:53.460772 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.460627    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d64520fa-e0cb-40f3-b404-499dbb851eb8" containerName="revoke-keys"
Jul 17 19:32:53.460772 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.460635    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d64520fa-e0cb-40f3-b404-499dbb851eb8" containerName="revoke-keys"
Jul 17 19:32:53.460772 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.460641    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d64520fa-e0cb-40f3-b404-499dbb851eb8" containerName="revoke-keys"
Jul 17 19:32:53.460772 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.460648    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fed73b5-5866-44c5-b044-b2661f39fce3" containerName="revoke-keys"
Jul 17 19:32:53.464766 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.464746    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh"
Jul 17 19:32:53.468471 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.468431    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j"]
Jul 17 19:32:53.473349 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.473266    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh"]
Jul 17 19:32:53.475624 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.475279    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-5gq2j"]
Jul 17 19:32:53.578922 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.578894    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7"
Jul 17 19:32:53.618826 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.618796    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9db11f5f-3290-4dfa-97bf-84ebcff69388-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh\" (UID: \"9db11f5f-3290-4dfa-97bf-84ebcff69388\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh"
Jul 17 19:32:53.720235 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.720154    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c66306b-9716-48f5-bca7-00dd1caed2cf-maas-api-service-ca\") pod \"1c66306b-9716-48f5-bca7-00dd1caed2cf\" (UID: \"1c66306b-9716-48f5-bca7-00dd1caed2cf\") "
Jul 17 19:32:53.720453 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.720437    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9db11f5f-3290-4dfa-97bf-84ebcff69388-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh\" (UID: \"9db11f5f-3290-4dfa-97bf-84ebcff69388\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh"
Jul 17 19:32:53.720546 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.720524    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c66306b-9716-48f5-bca7-00dd1caed2cf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c66306b-9716-48f5-bca7-00dd1caed2cf" (UID: "1c66306b-9716-48f5-bca7-00dd1caed2cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:32:53.721035 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.721018    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9db11f5f-3290-4dfa-97bf-84ebcff69388-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh\" (UID: \"9db11f5f-3290-4dfa-97bf-84ebcff69388\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh"
Jul 17 19:32:53.778282 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.778256    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh"
Jul 17 19:32:53.821838 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.821809    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c66306b-9716-48f5-bca7-00dd1caed2cf-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:32:53.912786 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:53.912761    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh"]
Jul 17 19:32:53.914419 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:32:53.914388    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db11f5f_3290_4dfa_97bf_84ebcff69388.slice/crio-49a8964fbe19d5e0b25a7abfbb08504a443d6a333dc5f5929022972d062029c6 WatchSource:0}: Error finding container 49a8964fbe19d5e0b25a7abfbb08504a443d6a333dc5f5929022972d062029c6: Status 404 returned error can't find the container with id 49a8964fbe19d5e0b25a7abfbb08504a443d6a333dc5f5929022972d062029c6
Jul 17 19:32:54.128563 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:54.128528    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d64520fa-e0cb-40f3-b404-499dbb851eb8" path="/var/lib/kubelet/pods/d64520fa-e0cb-40f3-b404-499dbb851eb8/volumes"
Jul 17 19:32:54.441346 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:54.441235    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh" event={"ID":"9db11f5f-3290-4dfa-97bf-84ebcff69388","Type":"ContainerStarted","Data":"8f4ac010fe5050ce3288fed67cc8d07efdb4ba048b38691a6a2ce1337cf507ff"}
Jul 17 19:32:54.441346 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:54.441326    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh" event={"ID":"9db11f5f-3290-4dfa-97bf-84ebcff69388","Type":"ContainerStarted","Data":"49a8964fbe19d5e0b25a7abfbb08504a443d6a333dc5f5929022972d062029c6"}
Jul 17 19:32:54.442654 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:54.442627    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7" event={"ID":"1c66306b-9716-48f5-bca7-00dd1caed2cf","Type":"ContainerDied","Data":"e22cdfe9999d8fbe8a7e1416e51801b9f45be750d38f89b2671fd9d523f69c7a"}
Jul 17 19:32:54.442783 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:54.442636    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7"
Jul 17 19:32:54.442783 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:54.442674    2559 scope.go:117] "RemoveContainer" containerID="3da5c215f668884255e0cbbb97e53b1cf75579ae30a7b7c38237a59641674de9"
Jul 17 19:32:54.457619 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:54.457575    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh" podStartSLOduration=1.457561946 podStartE2EDuration="1.457561946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:32:54.45569868 +0000 UTC m=+3584.897207370" watchObservedRunningTime="2026-07-17 19:32:54.457561946 +0000 UTC m=+3584.899070634"
Jul 17 19:32:54.468687 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:54.468666    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7"]
Jul 17 19:32:54.472408 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:54.472386    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-2wcz7"]
Jul 17 19:32:55.450621 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:55.450588    2559 generic.go:356] "Generic (PLEG): container finished" podID="9db11f5f-3290-4dfa-97bf-84ebcff69388" containerID="8f4ac010fe5050ce3288fed67cc8d07efdb4ba048b38691a6a2ce1337cf507ff" exitCode=6
Jul 17 19:32:55.451054 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:55.450665    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh" event={"ID":"9db11f5f-3290-4dfa-97bf-84ebcff69388","Type":"ContainerDied","Data":"8f4ac010fe5050ce3288fed67cc8d07efdb4ba048b38691a6a2ce1337cf507ff"}
Jul 17 19:32:55.451054 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:55.450821    2559 scope.go:117] "RemoveContainer" containerID="8f4ac010fe5050ce3288fed67cc8d07efdb4ba048b38691a6a2ce1337cf507ff"
Jul 17 19:32:56.126566 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:56.126534    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c66306b-9716-48f5-bca7-00dd1caed2cf" path="/var/lib/kubelet/pods/1c66306b-9716-48f5-bca7-00dd1caed2cf/volumes"
Jul 17 19:32:56.459210 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:56.459176    2559 generic.go:356] "Generic (PLEG): container finished" podID="9db11f5f-3290-4dfa-97bf-84ebcff69388" containerID="1d8eee59fedd0b55230ffac3945ae814b95511300de209925aec4caee415bfb3" exitCode=6
Jul 17 19:32:56.459664 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:56.459229    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh" event={"ID":"9db11f5f-3290-4dfa-97bf-84ebcff69388","Type":"ContainerDied","Data":"1d8eee59fedd0b55230ffac3945ae814b95511300de209925aec4caee415bfb3"}
Jul 17 19:32:56.459664 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:56.459284    2559 scope.go:117] "RemoveContainer" containerID="8f4ac010fe5050ce3288fed67cc8d07efdb4ba048b38691a6a2ce1337cf507ff"
Jul 17 19:32:56.459664 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:56.459478    2559 scope.go:117] "RemoveContainer" containerID="1d8eee59fedd0b55230ffac3945ae814b95511300de209925aec4caee415bfb3"
Jul 17 19:32:56.459827 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:32:56.459688    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh_odh-ai-gateway-infra(9db11f5f-3290-4dfa-97bf-84ebcff69388)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh" podUID="9db11f5f-3290-4dfa-97bf-84ebcff69388"
Jul 17 19:32:57.466610 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:32:57.466587    2559 scope.go:117] "RemoveContainer" containerID="1d8eee59fedd0b55230ffac3945ae814b95511300de209925aec4caee415bfb3"
Jul 17 19:32:57.466979 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:32:57.466772    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh_odh-ai-gateway-infra(9db11f5f-3290-4dfa-97bf-84ebcff69388)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh" podUID="9db11f5f-3290-4dfa-97bf-84ebcff69388"
Jul 17 19:33:01.119421 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:01.119335    2559 scope.go:117] "RemoveContainer" containerID="68f2d77498034f9c9338be503f390db28e752ff0dcb198029c5599ccfebae0a0"
Jul 17 19:33:01.492615 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:01.492583    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7" event={"ID":"ac95972f-340f-40b2-b781-3e8a18f9bf9e","Type":"ContainerStarted","Data":"5ca905b61d38d6c53175ee1607a9baf29dc57e84e161c8a7b9d865372f2162ae"}
Jul 17 19:33:01.509425 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:01.509378    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7" podStartSLOduration=17.509364167 podStartE2EDuration="17.509364167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:33:01.507512841 +0000 UTC m=+3591.949021530" watchObservedRunningTime="2026-07-17 19:33:01.509364167 +0000 UTC m=+3591.950872905"
Jul 17 19:33:02.499912 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:02.499879    2559 generic.go:356] "Generic (PLEG): container finished" podID="ac95972f-340f-40b2-b781-3e8a18f9bf9e" containerID="5ca905b61d38d6c53175ee1607a9baf29dc57e84e161c8a7b9d865372f2162ae" exitCode=6
Jul 17 19:33:02.500426 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:02.499952    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7" event={"ID":"ac95972f-340f-40b2-b781-3e8a18f9bf9e","Type":"ContainerDied","Data":"5ca905b61d38d6c53175ee1607a9baf29dc57e84e161c8a7b9d865372f2162ae"}
Jul 17 19:33:02.500426 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:02.500013    2559 scope.go:117] "RemoveContainer" containerID="68f2d77498034f9c9338be503f390db28e752ff0dcb198029c5599ccfebae0a0"
Jul 17 19:33:02.500426 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:02.500387    2559 scope.go:117] "RemoveContainer" containerID="5ca905b61d38d6c53175ee1607a9baf29dc57e84e161c8a7b9d865372f2162ae"
Jul 17 19:33:02.500669 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:33:02.500643    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7_odh-ai-gateway-infra(ac95972f-340f-40b2-b781-3e8a18f9bf9e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7" podUID="ac95972f-340f-40b2-b781-3e8a18f9bf9e"
Jul 17 19:33:02.523590 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:02.523551    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7"]
Jul 17 19:33:03.640733 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:03.640713    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7"
Jul 17 19:33:03.702681 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:03.702652    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac95972f-340f-40b2-b781-3e8a18f9bf9e-maas-api-service-ca\") pod \"ac95972f-340f-40b2-b781-3e8a18f9bf9e\" (UID: \"ac95972f-340f-40b2-b781-3e8a18f9bf9e\") "
Jul 17 19:33:03.702990 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:03.702967    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac95972f-340f-40b2-b781-3e8a18f9bf9e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac95972f-340f-40b2-b781-3e8a18f9bf9e" (UID: "ac95972f-340f-40b2-b781-3e8a18f9bf9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:33:03.803635 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:03.803572    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac95972f-340f-40b2-b781-3e8a18f9bf9e-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:33:04.514194 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:04.514112    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7"
Jul 17 19:33:04.514194 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:04.514123    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7" event={"ID":"ac95972f-340f-40b2-b781-3e8a18f9bf9e","Type":"ContainerDied","Data":"5e9dcc64af6863ece91166d7bb18c5b4925e085e40204b7c1a622c5169bbe449"}
Jul 17 19:33:04.514194 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:04.514176    2559 scope.go:117] "RemoveContainer" containerID="5ca905b61d38d6c53175ee1607a9baf29dc57e84e161c8a7b9d865372f2162ae"
Jul 17 19:33:04.532404 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:04.532377    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7"]
Jul 17 19:33:04.538773 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:04.538753    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0d4b3f70-ws2c7"]
Jul 17 19:33:06.130292 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:06.130252    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac95972f-340f-40b2-b781-3e8a18f9bf9e" path="/var/lib/kubelet/pods/ac95972f-340f-40b2-b781-3e8a18f9bf9e/volumes"
Jul 17 19:33:10.486987 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:10.486955    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 19:33:10.492490 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:10.492460    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 19:33:10.494650 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:10.494630    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 19:33:10.499738 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:10.499721    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhn5c_6d169050-68f4-4fff-abb6-b261d3c5935b/ovn-acl-logging/0.log"
Jul 17 19:33:11.119428 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.119343    2559 scope.go:117] "RemoveContainer" containerID="1d8eee59fedd0b55230ffac3945ae814b95511300de209925aec4caee415bfb3"
Jul 17 19:33:11.562321 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.562273    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh" event={"ID":"9db11f5f-3290-4dfa-97bf-84ebcff69388","Type":"ContainerStarted","Data":"366b36ed58953c18c635a36d214d542b1f559177738c7ac347fac5527180f1d3"}
Jul 17 19:33:11.826363 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.826295    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2109fd-lpbq7"]
Jul 17 19:33:11.826711 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.826699    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c66306b-9716-48f5-bca7-00dd1caed2cf" containerName="revoke-keys"
Jul 17 19:33:11.826772 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.826712    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c66306b-9716-48f5-bca7-00dd1caed2cf" containerName="revoke-keys"
Jul 17 19:33:11.826772 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.826727    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac95972f-340f-40b2-b781-3e8a18f9bf9e" containerName="revoke-keys"
Jul 17 19:33:11.826772 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.826733    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac95972f-340f-40b2-b781-3e8a18f9bf9e" containerName="revoke-keys"
Jul 17 19:33:11.826772 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.826744    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac95972f-340f-40b2-b781-3e8a18f9bf9e" containerName="revoke-keys"
Jul 17 19:33:11.826772 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.826750    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac95972f-340f-40b2-b781-3e8a18f9bf9e" containerName="revoke-keys"
Jul 17 19:33:11.826772 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.826760    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac95972f-340f-40b2-b781-3e8a18f9bf9e" containerName="revoke-keys"
Jul 17 19:33:11.826772 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.826766    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac95972f-340f-40b2-b781-3e8a18f9bf9e" containerName="revoke-keys"
Jul 17 19:33:11.826988 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.826779    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c66306b-9716-48f5-bca7-00dd1caed2cf" containerName="revoke-keys"
Jul 17 19:33:11.826988 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.826784    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c66306b-9716-48f5-bca7-00dd1caed2cf" containerName="revoke-keys"
Jul 17 19:33:11.826988 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.826854    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c66306b-9716-48f5-bca7-00dd1caed2cf" containerName="revoke-keys"
Jul 17 19:33:11.826988 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.826861    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac95972f-340f-40b2-b781-3e8a18f9bf9e" containerName="revoke-keys"
Jul 17 19:33:11.826988 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.826872    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c66306b-9716-48f5-bca7-00dd1caed2cf" containerName="revoke-keys"
Jul 17 19:33:11.826988 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.826877    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac95972f-340f-40b2-b781-3e8a18f9bf9e" containerName="revoke-keys"
Jul 17 19:33:11.826988 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.826883    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac95972f-340f-40b2-b781-3e8a18f9bf9e" containerName="revoke-keys"
Jul 17 19:33:11.830010 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.829988    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2109fd-lpbq7"
Jul 17 19:33:11.842693 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.842662    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2109fd-lpbq7"]
Jul 17 19:33:11.873578 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.873556    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ce51c0-e194-4364-b01f-88083324513c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-2109fd-lpbq7\" (UID: \"c6ce51c0-e194-4364-b01f-88083324513c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2109fd-lpbq7"
Jul 17 19:33:11.974422 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.974395    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ce51c0-e194-4364-b01f-88083324513c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-2109fd-lpbq7\" (UID: \"c6ce51c0-e194-4364-b01f-88083324513c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2109fd-lpbq7"
Jul 17 19:33:11.974926 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:11.974909    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ce51c0-e194-4364-b01f-88083324513c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-2109fd-lpbq7\" (UID: \"c6ce51c0-e194-4364-b01f-88083324513c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2109fd-lpbq7"
Jul 17 19:33:12.201929 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:12.201847    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2109fd-lpbq7"
Jul 17 19:33:12.339280 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:12.339252    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2109fd-lpbq7"]
Jul 17 19:33:12.340824 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:33:12.340798    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ce51c0_e194_4364_b01f_88083324513c.slice/crio-0cf2bf692b50afd78e1636f51822eac57c3b14cb5267fc46bcfe2ec46ccef0fd WatchSource:0}: Error finding container 0cf2bf692b50afd78e1636f51822eac57c3b14cb5267fc46bcfe2ec46ccef0fd: Status 404 returned error can't find the container with id 0cf2bf692b50afd78e1636f51822eac57c3b14cb5267fc46bcfe2ec46ccef0fd
Jul 17 19:33:12.569515 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:12.569485    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2109fd-lpbq7" event={"ID":"c6ce51c0-e194-4364-b01f-88083324513c","Type":"ContainerStarted","Data":"051e37ff0e5f438ed4fd1ff2c461afd60a9040a0c2510823451e272cc8541a50"}
Jul 17 19:33:12.569882 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:12.569546    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2109fd-lpbq7" event={"ID":"c6ce51c0-e194-4364-b01f-88083324513c","Type":"ContainerStarted","Data":"0cf2bf692b50afd78e1636f51822eac57c3b14cb5267fc46bcfe2ec46ccef0fd"}
Jul 17 19:33:12.571901 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:12.571881    2559 generic.go:356] "Generic (PLEG): container finished" podID="9db11f5f-3290-4dfa-97bf-84ebcff69388" containerID="366b36ed58953c18c635a36d214d542b1f559177738c7ac347fac5527180f1d3" exitCode=6
Jul 17 19:33:12.572015 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:12.571954    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh" event={"ID":"9db11f5f-3290-4dfa-97bf-84ebcff69388","Type":"ContainerDied","Data":"366b36ed58953c18c635a36d214d542b1f559177738c7ac347fac5527180f1d3"}
Jul 17 19:33:12.572015 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:12.572009    2559 scope.go:117] "RemoveContainer" containerID="1d8eee59fedd0b55230ffac3945ae814b95511300de209925aec4caee415bfb3"
Jul 17 19:33:12.572282 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:12.572267    2559 scope.go:117] "RemoveContainer" containerID="366b36ed58953c18c635a36d214d542b1f559177738c7ac347fac5527180f1d3"
Jul 17 19:33:12.572525 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:33:12.572504    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh_odh-ai-gateway-infra(9db11f5f-3290-4dfa-97bf-84ebcff69388)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh" podUID="9db11f5f-3290-4dfa-97bf-84ebcff69388"
Jul 17 19:33:12.586432 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:12.586393    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2109fd-lpbq7" podStartSLOduration=1.586380873 podStartE2EDuration="1.586380873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:33:12.584773334 +0000 UTC m=+3603.026282026" watchObservedRunningTime="2026-07-17 19:33:12.586380873 +0000 UTC m=+3603.027889561"
Jul 17 19:33:12.590764 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:12.590742    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh"]
Jul 17 19:33:13.581692 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:13.581666    2559 generic.go:356] "Generic (PLEG): container finished" podID="c6ce51c0-e194-4364-b01f-88083324513c" containerID="051e37ff0e5f438ed4fd1ff2c461afd60a9040a0c2510823451e272cc8541a50" exitCode=0
Jul 17 19:33:13.582023 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:13.581705    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2109fd-lpbq7" event={"ID":"c6ce51c0-e194-4364-b01f-88083324513c","Type":"ContainerDied","Data":"051e37ff0e5f438ed4fd1ff2c461afd60a9040a0c2510823451e272cc8541a50"}
Jul 17 19:33:13.707637 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:13.707614    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh"
Jul 17 19:33:13.793042 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:13.793009    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9db11f5f-3290-4dfa-97bf-84ebcff69388-maas-api-service-ca\") pod \"9db11f5f-3290-4dfa-97bf-84ebcff69388\" (UID: \"9db11f5f-3290-4dfa-97bf-84ebcff69388\") "
Jul 17 19:33:13.793365 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:13.793345    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db11f5f-3290-4dfa-97bf-84ebcff69388-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9db11f5f-3290-4dfa-97bf-84ebcff69388" (UID: "9db11f5f-3290-4dfa-97bf-84ebcff69388"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:33:13.894236 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:13.894176    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9db11f5f-3290-4dfa-97bf-84ebcff69388-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:33:14.588440 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:14.588413    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh"
Jul 17 19:33:14.588850 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:14.588446    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh" event={"ID":"9db11f5f-3290-4dfa-97bf-84ebcff69388","Type":"ContainerDied","Data":"49a8964fbe19d5e0b25a7abfbb08504a443d6a333dc5f5929022972d062029c6"}
Jul 17 19:33:14.588850 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:14.588497    2559 scope.go:117] "RemoveContainer" containerID="366b36ed58953c18c635a36d214d542b1f559177738c7ac347fac5527180f1d3"
Jul 17 19:33:14.605783 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:14.605758    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh"]
Jul 17 19:33:14.609546 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:14.609521    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34da1ffa-7r4rh"]
Jul 17 19:33:14.723012 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:14.722991    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2109fd-lpbq7"
Jul 17 19:33:14.802064 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:14.802041    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ce51c0-e194-4364-b01f-88083324513c-maas-api-service-ca\") pod \"c6ce51c0-e194-4364-b01f-88083324513c\" (UID: \"c6ce51c0-e194-4364-b01f-88083324513c\") "
Jul 17 19:33:14.802414 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:14.802390    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ce51c0-e194-4364-b01f-88083324513c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6ce51c0-e194-4364-b01f-88083324513c" (UID: "c6ce51c0-e194-4364-b01f-88083324513c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:33:14.902800 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:14.902733    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ce51c0-e194-4364-b01f-88083324513c-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:33:15.596358 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:15.596323    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2109fd-lpbq7"
Jul 17 19:33:15.596358 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:15.596335    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2109fd-lpbq7" event={"ID":"c6ce51c0-e194-4364-b01f-88083324513c","Type":"ContainerDied","Data":"0cf2bf692b50afd78e1636f51822eac57c3b14cb5267fc46bcfe2ec46ccef0fd"}
Jul 17 19:33:15.596832 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:15.596380    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cf2bf692b50afd78e1636f51822eac57c3b14cb5267fc46bcfe2ec46ccef0fd"
Jul 17 19:33:16.125105 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:16.125079    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9db11f5f-3290-4dfa-97bf-84ebcff69388" path="/var/lib/kubelet/pods/9db11f5f-3290-4dfa-97bf-84ebcff69388/volumes"
Jul 17 19:33:26.433079 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.433045    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf"]
Jul 17 19:33:26.433474 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.433432    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6ce51c0-e194-4364-b01f-88083324513c" containerName="revoke-keys"
Jul 17 19:33:26.433474 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.433445    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6ce51c0-e194-4364-b01f-88083324513c" containerName="revoke-keys"
Jul 17 19:33:26.433474 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.433458    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9db11f5f-3290-4dfa-97bf-84ebcff69388" containerName="revoke-keys"
Jul 17 19:33:26.433474 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.433463    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9db11f5f-3290-4dfa-97bf-84ebcff69388" containerName="revoke-keys"
Jul 17 19:33:26.433474 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.433475    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9db11f5f-3290-4dfa-97bf-84ebcff69388" containerName="revoke-keys"
Jul 17 19:33:26.433633 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.433480    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9db11f5f-3290-4dfa-97bf-84ebcff69388" containerName="revoke-keys"
Jul 17 19:33:26.433633 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.433493    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c66306b-9716-48f5-bca7-00dd1caed2cf" containerName="revoke-keys"
Jul 17 19:33:26.433633 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.433498    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c66306b-9716-48f5-bca7-00dd1caed2cf" containerName="revoke-keys"
Jul 17 19:33:26.433633 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.433506    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9db11f5f-3290-4dfa-97bf-84ebcff69388" containerName="revoke-keys"
Jul 17 19:33:26.433633 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.433511    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9db11f5f-3290-4dfa-97bf-84ebcff69388" containerName="revoke-keys"
Jul 17 19:33:26.433633 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.433587    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c66306b-9716-48f5-bca7-00dd1caed2cf" containerName="revoke-keys"
Jul 17 19:33:26.433633 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.433599    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9db11f5f-3290-4dfa-97bf-84ebcff69388" containerName="revoke-keys"
Jul 17 19:33:26.433633 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.433604    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9db11f5f-3290-4dfa-97bf-84ebcff69388" containerName="revoke-keys"
Jul 17 19:33:26.433633 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.433609    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9db11f5f-3290-4dfa-97bf-84ebcff69388" containerName="revoke-keys"
Jul 17 19:33:26.433633 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.433615    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6ce51c0-e194-4364-b01f-88083324513c" containerName="revoke-keys"
Jul 17 19:33:26.436787 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.436770    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf"
Jul 17 19:33:26.439702 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.439680    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\""
Jul 17 19:33:26.444479 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.444435    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf"]
Jul 17 19:33:26.494226 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.494189    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcad4203-5c10-42d4-8445-fb8fef2383ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf\" (UID: \"fcad4203-5c10-42d4-8445-fb8fef2383ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf"
Jul 17 19:33:26.595552 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.595516    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcad4203-5c10-42d4-8445-fb8fef2383ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf\" (UID: \"fcad4203-5c10-42d4-8445-fb8fef2383ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf"
Jul 17 19:33:26.596121 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.596098    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcad4203-5c10-42d4-8445-fb8fef2383ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf\" (UID: \"fcad4203-5c10-42d4-8445-fb8fef2383ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf"
Jul 17 19:33:26.749461 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.749382    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf"
Jul 17 19:33:26.889041 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:26.889015    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf"]
Jul 17 19:33:26.891245 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:33:26.891220    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcad4203_5c10_42d4_8445_fb8fef2383ba.slice/crio-179311dd195d1e45de60b16f3baecb2d8539020b20ed07b6345b60e6a1a1ecdf WatchSource:0}: Error finding container 179311dd195d1e45de60b16f3baecb2d8539020b20ed07b6345b60e6a1a1ecdf: Status 404 returned error can't find the container with id 179311dd195d1e45de60b16f3baecb2d8539020b20ed07b6345b60e6a1a1ecdf
Jul 17 19:33:27.672098 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:27.672069    2559 generic.go:356] "Generic (PLEG): container finished" podID="fcad4203-5c10-42d4-8445-fb8fef2383ba" containerID="8fe19716cc1139890a520ecd9079f91fca89cc028be9e4b714976af756a04194" exitCode=6
Jul 17 19:33:27.672503 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:27.672119    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf" event={"ID":"fcad4203-5c10-42d4-8445-fb8fef2383ba","Type":"ContainerDied","Data":"8fe19716cc1139890a520ecd9079f91fca89cc028be9e4b714976af756a04194"}
Jul 17 19:33:27.672503 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:27.672155    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf" event={"ID":"fcad4203-5c10-42d4-8445-fb8fef2383ba","Type":"ContainerStarted","Data":"179311dd195d1e45de60b16f3baecb2d8539020b20ed07b6345b60e6a1a1ecdf"}
Jul 17 19:33:27.672503 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:27.672394    2559 scope.go:117] "RemoveContainer" containerID="8fe19716cc1139890a520ecd9079f91fca89cc028be9e4b714976af756a04194"
Jul 17 19:33:28.680609 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:28.680526    2559 generic.go:356] "Generic (PLEG): container finished" podID="fcad4203-5c10-42d4-8445-fb8fef2383ba" containerID="b5f577d8753e9a307b3460102e1115f8b114c590415c974b860968238374e5ac" exitCode=6
Jul 17 19:33:28.680609 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:28.680582    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf" event={"ID":"fcad4203-5c10-42d4-8445-fb8fef2383ba","Type":"ContainerDied","Data":"b5f577d8753e9a307b3460102e1115f8b114c590415c974b860968238374e5ac"}
Jul 17 19:33:28.681072 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:28.680632    2559 scope.go:117] "RemoveContainer" containerID="8fe19716cc1139890a520ecd9079f91fca89cc028be9e4b714976af756a04194"
Jul 17 19:33:28.681072 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:28.680830    2559 scope.go:117] "RemoveContainer" containerID="b5f577d8753e9a307b3460102e1115f8b114c590415c974b860968238374e5ac"
Jul 17 19:33:28.681158 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:33:28.681080    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf_odh-ai-gateway-infra(fcad4203-5c10-42d4-8445-fb8fef2383ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf" podUID="fcad4203-5c10-42d4-8445-fb8fef2383ba"
Jul 17 19:33:29.688949 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:29.688922    2559 scope.go:117] "RemoveContainer" containerID="b5f577d8753e9a307b3460102e1115f8b114c590415c974b860968238374e5ac"
Jul 17 19:33:29.689405 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:33:29.689095    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf_odh-ai-gateway-infra(fcad4203-5c10-42d4-8445-fb8fef2383ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf" podUID="fcad4203-5c10-42d4-8445-fb8fef2383ba"
Jul 17 19:33:31.727066 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:31.727033    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz"]
Jul 17 19:33:31.731648 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:31.731632    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz"
Jul 17 19:33:31.736449 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:31.736419    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz"]
Jul 17 19:33:31.835736 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:31.835700    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz\" (UID: \"60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz"
Jul 17 19:33:31.936578 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:31.936546    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz\" (UID: \"60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz"
Jul 17 19:33:31.937087 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:31.937070    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz\" (UID: \"60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz"
Jul 17 19:33:32.043785 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:32.043725    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz"
Jul 17 19:33:32.177750 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:32.177722    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz"]
Jul 17 19:33:32.179273 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:33:32.179253    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f1c8c0_1e28_4d2e_b44a_16bc510fbcb6.slice/crio-1b4164c5530cfa970fde6802e8fcd702eeaec6ea1df976fbadf9d48698de95b2 WatchSource:0}: Error finding container 1b4164c5530cfa970fde6802e8fcd702eeaec6ea1df976fbadf9d48698de95b2: Status 404 returned error can't find the container with id 1b4164c5530cfa970fde6802e8fcd702eeaec6ea1df976fbadf9d48698de95b2
Jul 17 19:33:32.708134 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:32.708051    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz" event={"ID":"60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6","Type":"ContainerStarted","Data":"979115b3c132bf275cdfae4665dbca11f4284f09345001513a2af5717bacd8e2"}
Jul 17 19:33:32.708134 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:32.708106    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz" event={"ID":"60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6","Type":"ContainerStarted","Data":"1b4164c5530cfa970fde6802e8fcd702eeaec6ea1df976fbadf9d48698de95b2"}
Jul 17 19:33:32.724096 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:32.724050    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz" podStartSLOduration=1.724035902 podStartE2EDuration="1.724035902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:33:32.722623258 +0000 UTC m=+3623.164131951" watchObservedRunningTime="2026-07-17 19:33:32.724035902 +0000 UTC m=+3623.165544591"
Jul 17 19:33:33.716378 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:33.716337    2559 generic.go:356] "Generic (PLEG): container finished" podID="60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6" containerID="979115b3c132bf275cdfae4665dbca11f4284f09345001513a2af5717bacd8e2" exitCode=6
Jul 17 19:33:33.716781 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:33.716417    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz" event={"ID":"60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6","Type":"ContainerDied","Data":"979115b3c132bf275cdfae4665dbca11f4284f09345001513a2af5717bacd8e2"}
Jul 17 19:33:33.716781 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:33.716572    2559 scope.go:117] "RemoveContainer" containerID="979115b3c132bf275cdfae4665dbca11f4284f09345001513a2af5717bacd8e2"
Jul 17 19:33:34.726598 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:34.726512    2559 generic.go:356] "Generic (PLEG): container finished" podID="60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6" containerID="8974b8349b1e1bf5276ca2c2430eb30ff58e7045db06603e36f9d1d7848fd6ca" exitCode=6
Jul 17 19:33:34.726598 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:34.726572    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz" event={"ID":"60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6","Type":"ContainerDied","Data":"8974b8349b1e1bf5276ca2c2430eb30ff58e7045db06603e36f9d1d7848fd6ca"}
Jul 17 19:33:34.727060 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:34.726630    2559 scope.go:117] "RemoveContainer" containerID="979115b3c132bf275cdfae4665dbca11f4284f09345001513a2af5717bacd8e2"
Jul 17 19:33:34.727060 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:34.726850    2559 scope.go:117] "RemoveContainer" containerID="8974b8349b1e1bf5276ca2c2430eb30ff58e7045db06603e36f9d1d7848fd6ca"
Jul 17 19:33:34.727187 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:33:34.727094    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz_odh-ai-gateway-infra(60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz" podUID="60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6"
Jul 17 19:33:35.735784 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:35.735756    2559 scope.go:117] "RemoveContainer" containerID="8974b8349b1e1bf5276ca2c2430eb30ff58e7045db06603e36f9d1d7848fd6ca"
Jul 17 19:33:35.736201 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:33:35.735925    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz_odh-ai-gateway-infra(60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz" podUID="60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6"
Jul 17 19:33:41.119325 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:41.119282    2559 scope.go:117] "RemoveContainer" containerID="b5f577d8753e9a307b3460102e1115f8b114c590415c974b860968238374e5ac"
Jul 17 19:33:41.781284 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:41.781208    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf" event={"ID":"fcad4203-5c10-42d4-8445-fb8fef2383ba","Type":"ContainerStarted","Data":"f01987af094ec2a3091b8f5ab385c37cac2c411b714e081fffded435522ea07d"}
Jul 17 19:33:41.799751 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:41.799705    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf" podStartSLOduration=15.799691004 podStartE2EDuration="15.799691004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:33:41.797743878 +0000 UTC m=+3632.239252569" watchObservedRunningTime="2026-07-17 19:33:41.799691004 +0000 UTC m=+3632.241199693"
Jul 17 19:33:42.790543 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:42.790506    2559 generic.go:356] "Generic (PLEG): container finished" podID="fcad4203-5c10-42d4-8445-fb8fef2383ba" containerID="f01987af094ec2a3091b8f5ab385c37cac2c411b714e081fffded435522ea07d" exitCode=6
Jul 17 19:33:42.790959 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:42.790582    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf" event={"ID":"fcad4203-5c10-42d4-8445-fb8fef2383ba","Type":"ContainerDied","Data":"f01987af094ec2a3091b8f5ab385c37cac2c411b714e081fffded435522ea07d"}
Jul 17 19:33:42.790959 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:42.790645    2559 scope.go:117] "RemoveContainer" containerID="b5f577d8753e9a307b3460102e1115f8b114c590415c974b860968238374e5ac"
Jul 17 19:33:42.791093 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:42.790964    2559 scope.go:117] "RemoveContainer" containerID="f01987af094ec2a3091b8f5ab385c37cac2c411b714e081fffded435522ea07d"
Jul 17 19:33:42.791239 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:33:42.791214    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf_odh-ai-gateway-infra(fcad4203-5c10-42d4-8445-fb8fef2383ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf" podUID="fcad4203-5c10-42d4-8445-fb8fef2383ba"
Jul 17 19:33:42.808203 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:42.808181    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf"]
Jul 17 19:33:43.760445 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:43.760403    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7c84488dd8-2nhq2_94530b49-743c-4bac-812f-897823b999b2/manager/0.log"
Jul 17 19:33:43.929922 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:43.929894    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf"
Jul 17 19:33:44.035847 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:44.035774    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcad4203-5c10-42d4-8445-fb8fef2383ba-maas-api-service-ca\") pod \"fcad4203-5c10-42d4-8445-fb8fef2383ba\" (UID: \"fcad4203-5c10-42d4-8445-fb8fef2383ba\") "
Jul 17 19:33:44.036131 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:44.036109    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcad4203-5c10-42d4-8445-fb8fef2383ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fcad4203-5c10-42d4-8445-fb8fef2383ba" (UID: "fcad4203-5c10-42d4-8445-fb8fef2383ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:33:44.137131 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:44.137108    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcad4203-5c10-42d4-8445-fb8fef2383ba-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:33:44.370249 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:44.370230    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57479f7bfb-xfmhk_101e4b60-dfa6-48e7-97e1-0de5c935dbef/copy-manifests/0.log"
Jul 17 19:33:44.382886 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:44.382861    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57479f7bfb-xfmhk_101e4b60-dfa6-48e7-97e1-0de5c935dbef/manager/0.log"
Jul 17 19:33:44.731815 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:44.731758    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-c574cb76b-x9kng_69778d64-192f-4acb-a18c-9b849d7b00ce/maas-api/0.log"
Jul 17 19:33:44.805298 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:44.805271    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf" event={"ID":"fcad4203-5c10-42d4-8445-fb8fef2383ba","Type":"ContainerDied","Data":"179311dd195d1e45de60b16f3baecb2d8539020b20ed07b6345b60e6a1a1ecdf"}
Jul 17 19:33:44.805402 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:44.805351    2559 scope.go:117] "RemoveContainer" containerID="f01987af094ec2a3091b8f5ab385c37cac2c411b714e081fffded435522ea07d"
Jul 17 19:33:44.805402 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:44.805278    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf"
Jul 17 19:33:44.823089 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:44.823066    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf"]
Jul 17 19:33:44.829107 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:44.829082    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-t7hqf"]
Jul 17 19:33:44.845870 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:44.845843    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738580-5ppb8_7e69000d-50b6-4f0e-8e42-d56be0992b71/cleanup/0.log"
Jul 17 19:33:44.953018 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:44.952999    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738595-hwqmv_dfc4998a-504f-4a3f-af90-8d5f03d500a3/cleanup/0.log"
Jul 17 19:33:45.280585 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:45.280557    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-40565f-wthwq_2f0d0c4a-928a-488b-be6d-0c274a3c604a/revoke-keys/0.log"
Jul 17 19:33:45.393718 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:45.393693    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-460276-fjvg4_c94f553d-0c14-415b-97a8-ac1ec371f547/revoke-keys/0.log"
Jul 17 19:33:45.502343 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:45.502299    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz_60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6/revoke-keys/1.log"
Jul 17 19:33:45.715981 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:45.715954    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-15f29d-7m5wl_fe5ba99a-ce82-48e2-9f58-ce45b0e2315a/revoke-keys/0.log"
Jul 17 19:33:45.932370 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:45.932348    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-010d52ef-qbjc2_3e01a532-d33d-4def-bb24-4b1386a5aa3b/revoke-keys/0.log"
Jul 17 19:33:46.039041 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:46.038984    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-bd0f3bb1-pp8qd_f433815f-03d0-4019-955a-20d4b31fa600/revoke-keys/0.log"
Jul 17 19:33:46.125244 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:46.125216    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcad4203-5c10-42d4-8445-fb8fef2383ba" path="/var/lib/kubelet/pods/fcad4203-5c10-42d4-8445-fb8fef2383ba/volumes"
Jul 17 19:33:46.150143 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:46.150119    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-98e305-zkxlm_6fed73b5-5866-44c5-b044-b2661f39fce3/revoke-keys/0.log"
Jul 17 19:33:46.257454 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:46.257432    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-2109fd-lpbq7_c6ce51c0-e194-4364-b01f-88083324513c/revoke-keys/0.log"
Jul 17 19:33:46.371526 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:46.371508    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-qcm76_7aba07c7-e794-46d0-887a-0171d5250c5f/postgres/0.log"
Jul 17 19:33:47.111696 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:47.111674    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc_1dc407dc-206b-44f2-9934-19d747560cb4/util/0.log"
Jul 17 19:33:47.118041 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:47.118017    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc_1dc407dc-206b-44f2-9934-19d747560cb4/pull/0.log"
Jul 17 19:33:47.124091 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:47.124075    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc_1dc407dc-206b-44f2-9934-19d747560cb4/extract/0.log"
Jul 17 19:33:47.234500 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:47.234483    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29_bfb82736-cf62-4f65-b8ae-d7f58162fce6/util/0.log"
Jul 17 19:33:47.240873 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:47.240855    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29_bfb82736-cf62-4f65-b8ae-d7f58162fce6/pull/0.log"
Jul 17 19:33:47.247381 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:47.247364    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29_bfb82736-cf62-4f65-b8ae-d7f58162fce6/extract/0.log"
Jul 17 19:33:47.352050 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:47.352033    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds_d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5/util/0.log"
Jul 17 19:33:47.357853 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:47.357836    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds_d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5/pull/0.log"
Jul 17 19:33:47.363377 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:47.363322    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds_d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5/extract/0.log"
Jul 17 19:33:47.473958 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:47.473937    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64_6624d4e9-b35a-4073-bda3-4dbf001ba7e0/pull/0.log"
Jul 17 19:33:47.480492 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:47.480475    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64_6624d4e9-b35a-4073-bda3-4dbf001ba7e0/extract/0.log"
Jul 17 19:33:47.486610 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:47.486588    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64_6624d4e9-b35a-4073-bda3-4dbf001ba7e0/util/0.log"
Jul 17 19:33:47.924703 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:47.924679    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-pqdnf_9138212c-21f1-414e-a2c3-d726e2d8b609/kuadrant-console-plugin/0.log"
Jul 17 19:33:49.086279 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:49.086254    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864488b8fd-fxtq4_367eed95-dbb9-47d7-aa40-bc764b363849/kube-auth-proxy/0.log"
Jul 17 19:33:49.119491 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:49.119468    2559 scope.go:117] "RemoveContainer" containerID="8974b8349b1e1bf5276ca2c2430eb30ff58e7045db06603e36f9d1d7848fd6ca"
Jul 17 19:33:49.839441 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:49.839416    2559 generic.go:356] "Generic (PLEG): container finished" podID="60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6" containerID="ffe98b8281cb1307bbfb763a619d17087e76346efb943bc8b8a73f77c4155650" exitCode=6
Jul 17 19:33:49.839558 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:49.839480    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz" event={"ID":"60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6","Type":"ContainerDied","Data":"ffe98b8281cb1307bbfb763a619d17087e76346efb943bc8b8a73f77c4155650"}
Jul 17 19:33:49.839558 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:49.839537    2559 scope.go:117] "RemoveContainer" containerID="8974b8349b1e1bf5276ca2c2430eb30ff58e7045db06603e36f9d1d7848fd6ca"
Jul 17 19:33:49.839845 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:49.839824    2559 scope.go:117] "RemoveContainer" containerID="ffe98b8281cb1307bbfb763a619d17087e76346efb943bc8b8a73f77c4155650"
Jul 17 19:33:49.840043 ip-10-0-138-51 kubenswrapper[2559]: E0717 19:33:49.840011    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz_odh-ai-gateway-infra(60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz" podUID="60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6"
Jul 17 19:33:49.892021 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:49.891995    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-4zcdj_67ebe5a2-2fa1-449a-a873-b551f2cfcc7b/main/0.log"
Jul 17 19:33:50.226982 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:50.226956    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsnwc6_3e6fdeec-4355-445a-9d68-aca800edc6e6/main/0.log"
Jul 17 19:33:50.865856 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:50.865824    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz"]
Jul 17 19:33:50.996586 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:50.996562    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz"
Jul 17 19:33:51.093337 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:51.093275    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6-maas-api-service-ca\") pod \"60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6\" (UID: \"60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6\") "
Jul 17 19:33:51.093601 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:51.093578    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6" (UID: "60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:33:51.194519 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:51.194446    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6-maas-api-service-ca\") on node \"ip-10-0-138-51.ec2.internal\" DevicePath \"\""
Jul 17 19:33:51.853780 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:51.853753    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz"
Jul 17 19:33:51.854201 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:51.853753    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz" event={"ID":"60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6","Type":"ContainerDied","Data":"1b4164c5530cfa970fde6802e8fcd702eeaec6ea1df976fbadf9d48698de95b2"}
Jul 17 19:33:51.854201 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:51.853883    2559 scope.go:117] "RemoveContainer" containerID="ffe98b8281cb1307bbfb763a619d17087e76346efb943bc8b8a73f77c4155650"
Jul 17 19:33:51.877845 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:51.877819    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz"]
Jul 17 19:33:51.883357 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:51.883335    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-q9tsz"]
Jul 17 19:33:52.126079 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:52.126011    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6" path="/var/lib/kubelet/pods/60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6/volumes"
Jul 17 19:33:56.032618 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.032582    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dtqk6/must-gather-7qh5q"]
Jul 17 19:33:56.033019 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.032975    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcad4203-5c10-42d4-8445-fb8fef2383ba" containerName="revoke-keys"
Jul 17 19:33:56.033019 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.032986    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcad4203-5c10-42d4-8445-fb8fef2383ba" containerName="revoke-keys"
Jul 17 19:33:56.033019 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.032996    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcad4203-5c10-42d4-8445-fb8fef2383ba" containerName="revoke-keys"
Jul 17 19:33:56.033019 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.033002    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcad4203-5c10-42d4-8445-fb8fef2383ba" containerName="revoke-keys"
Jul 17 19:33:56.033019 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.033019    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6" containerName="revoke-keys"
Jul 17 19:33:56.033214 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.033025    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6" containerName="revoke-keys"
Jul 17 19:33:56.033214 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.033032    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6" containerName="revoke-keys"
Jul 17 19:33:56.033214 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.033037    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6" containerName="revoke-keys"
Jul 17 19:33:56.033214 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.033048    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6" containerName="revoke-keys"
Jul 17 19:33:56.033214 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.033054    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6" containerName="revoke-keys"
Jul 17 19:33:56.033214 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.033110    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcad4203-5c10-42d4-8445-fb8fef2383ba" containerName="revoke-keys"
Jul 17 19:33:56.033214 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.033117    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcad4203-5c10-42d4-8445-fb8fef2383ba" containerName="revoke-keys"
Jul 17 19:33:56.033214 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.033124    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6" containerName="revoke-keys"
Jul 17 19:33:56.033214 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.033140    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6" containerName="revoke-keys"
Jul 17 19:33:56.033214 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.033147    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcad4203-5c10-42d4-8445-fb8fef2383ba" containerName="revoke-keys"
Jul 17 19:33:56.033214 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.033154    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f1c8c0-1e28-4d2e-b44a-16bc510fbcb6" containerName="revoke-keys"
Jul 17 19:33:56.033656 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.033221    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcad4203-5c10-42d4-8445-fb8fef2383ba" containerName="revoke-keys"
Jul 17 19:33:56.033656 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.033227    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcad4203-5c10-42d4-8445-fb8fef2383ba" containerName="revoke-keys"
Jul 17 19:33:56.036559 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.036541    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dtqk6/must-gather-7qh5q"
Jul 17 19:33:56.039666 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.039633    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dtqk6\"/\"kube-root-ca.crt\""
Jul 17 19:33:56.039666 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.039653    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dtqk6\"/\"openshift-service-ca.crt\""
Jul 17 19:33:56.044557 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.044533    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dtqk6/must-gather-7qh5q"]
Jul 17 19:33:56.137445 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.137418    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e0dd4efc-bc98-4f93-80ee-4480c4ba6449-must-gather-output\") pod \"must-gather-7qh5q\" (UID: \"e0dd4efc-bc98-4f93-80ee-4480c4ba6449\") " pod="openshift-must-gather-dtqk6/must-gather-7qh5q"
Jul 17 19:33:56.137559 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.137499    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv5h9\" (UniqueName: \"kubernetes.io/projected/e0dd4efc-bc98-4f93-80ee-4480c4ba6449-kube-api-access-vv5h9\") pod \"must-gather-7qh5q\" (UID: \"e0dd4efc-bc98-4f93-80ee-4480c4ba6449\") " pod="openshift-must-gather-dtqk6/must-gather-7qh5q"
Jul 17 19:33:56.238484 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.238456    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv5h9\" (UniqueName: \"kubernetes.io/projected/e0dd4efc-bc98-4f93-80ee-4480c4ba6449-kube-api-access-vv5h9\") pod \"must-gather-7qh5q\" (UID: \"e0dd4efc-bc98-4f93-80ee-4480c4ba6449\") " pod="openshift-must-gather-dtqk6/must-gather-7qh5q"
Jul 17 19:33:56.238589 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.238511    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e0dd4efc-bc98-4f93-80ee-4480c4ba6449-must-gather-output\") pod \"must-gather-7qh5q\" (UID: \"e0dd4efc-bc98-4f93-80ee-4480c4ba6449\") " pod="openshift-must-gather-dtqk6/must-gather-7qh5q"
Jul 17 19:33:56.238783 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.238768    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e0dd4efc-bc98-4f93-80ee-4480c4ba6449-must-gather-output\") pod \"must-gather-7qh5q\" (UID: \"e0dd4efc-bc98-4f93-80ee-4480c4ba6449\") " pod="openshift-must-gather-dtqk6/must-gather-7qh5q"
Jul 17 19:33:56.246957 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.246928    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv5h9\" (UniqueName: \"kubernetes.io/projected/e0dd4efc-bc98-4f93-80ee-4480c4ba6449-kube-api-access-vv5h9\") pod \"must-gather-7qh5q\" (UID: \"e0dd4efc-bc98-4f93-80ee-4480c4ba6449\") " pod="openshift-must-gather-dtqk6/must-gather-7qh5q"
Jul 17 19:33:56.347883 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.347862    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dtqk6/must-gather-7qh5q"
Jul 17 19:33:56.483595 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.483564    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dtqk6/must-gather-7qh5q"]
Jul 17 19:33:56.484737 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:33:56.484710    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0dd4efc_bc98_4f93_80ee_4480c4ba6449.slice/crio-9a09086efff031568247488810694ba1535faac4b60ccfe43a8a077f3662126c WatchSource:0}: Error finding container 9a09086efff031568247488810694ba1535faac4b60ccfe43a8a077f3662126c: Status 404 returned error can't find the container with id 9a09086efff031568247488810694ba1535faac4b60ccfe43a8a077f3662126c
Jul 17 19:33:56.486539 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.486526    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 19:33:56.887645 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:56.887617    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dtqk6/must-gather-7qh5q" event={"ID":"e0dd4efc-bc98-4f93-80ee-4480c4ba6449","Type":"ContainerStarted","Data":"9a09086efff031568247488810694ba1535faac4b60ccfe43a8a077f3662126c"}
Jul 17 19:33:57.897980 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:57.897948    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dtqk6/must-gather-7qh5q" event={"ID":"e0dd4efc-bc98-4f93-80ee-4480c4ba6449","Type":"ContainerStarted","Data":"24f501eae0626fddca72621181447faa35c9589b442fe161422f333ef4cd0bcd"}
Jul 17 19:33:57.898393 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:57.897989    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dtqk6/must-gather-7qh5q" event={"ID":"e0dd4efc-bc98-4f93-80ee-4480c4ba6449","Type":"ContainerStarted","Data":"19074ea62a9bd86f31fb02afe98e9b6904c43c8909a8b951e7e28302cb71e72e"}
Jul 17 19:33:57.913722 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:57.913671    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dtqk6/must-gather-7qh5q" podStartSLOduration=1.175084166 podStartE2EDuration="1.913654058s" totalImagesPullingTime="738.569892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:33:57.91152326 +0000 UTC m=+3648.353031950" watchObservedRunningTime="2026-07-17 19:33:57.913654058 +0000 UTC m=+3648.355162746"
Jul 17 19:33:58.779480 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:58.779451    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5nws8_1f8b093e-7ab9-406d-8a0e-c79e694cb205/global-pull-secret-syncer/0.log"
Jul 17 19:33:59.006702 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:59.006672    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vtswf_e35f12b9-1b2e-474f-8261-ec3e25a25775/konnectivity-agent/0.log"
Jul 17 19:33:59.078705 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:33:59.078677    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-51.ec2.internal_68a4b3cbdf87f36c033c410a5f90d655/haproxy/0.log"
Jul 17 19:34:03.300046 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:03.299981    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc_1dc407dc-206b-44f2-9934-19d747560cb4/extract/0.log"
Jul 17 19:34:03.323486 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:03.323392    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc_1dc407dc-206b-44f2-9934-19d747560cb4/util/0.log"
Jul 17 19:34:03.343638 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:03.343613    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gbkrc_1dc407dc-206b-44f2-9934-19d747560cb4/pull/0.log"
Jul 17 19:34:03.379992 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:03.379967    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29_bfb82736-cf62-4f65-b8ae-d7f58162fce6/extract/0.log"
Jul 17 19:34:03.401535 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:03.401499    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29_bfb82736-cf62-4f65-b8ae-d7f58162fce6/util/0.log"
Jul 17 19:34:03.425819 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:03.425793    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlv29_bfb82736-cf62-4f65-b8ae-d7f58162fce6/pull/0.log"
Jul 17 19:34:03.451151 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:03.451127    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds_d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5/extract/0.log"
Jul 17 19:34:03.480551 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:03.480527    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds_d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5/util/0.log"
Jul 17 19:34:03.502591 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:03.502554    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73879ds_d83d8a2a-28b5-4efd-b344-dbab5ddbb9b5/pull/0.log"
Jul 17 19:34:03.530178 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:03.530130    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64_6624d4e9-b35a-4073-bda3-4dbf001ba7e0/extract/0.log"
Jul 17 19:34:03.551851 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:03.551766    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64_6624d4e9-b35a-4073-bda3-4dbf001ba7e0/util/0.log"
Jul 17 19:34:03.573409 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:03.573377    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12mn64_6624d4e9-b35a-4073-bda3-4dbf001ba7e0/pull/0.log"
Jul 17 19:34:04.296812 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:04.296785    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-pqdnf_9138212c-21f1-414e-a2c3-d726e2d8b609/kuadrant-console-plugin/0.log"
Jul 17 19:34:06.392532 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:06.392498    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8c076bd6-5c9d-4dc8-93de-e54c69cf8673/alertmanager/0.log"
Jul 17 19:34:06.417860 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:06.417826    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8c076bd6-5c9d-4dc8-93de-e54c69cf8673/config-reloader/0.log"
Jul 17 19:34:06.440433 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:06.440408    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8c076bd6-5c9d-4dc8-93de-e54c69cf8673/kube-rbac-proxy-web/0.log"
Jul 17 19:34:06.465194 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:06.465170    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8c076bd6-5c9d-4dc8-93de-e54c69cf8673/kube-rbac-proxy/0.log"
Jul 17 19:34:06.492241 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:06.492195    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8c076bd6-5c9d-4dc8-93de-e54c69cf8673/kube-rbac-proxy-metric/0.log"
Jul 17 19:34:06.515495 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:06.515467    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8c076bd6-5c9d-4dc8-93de-e54c69cf8673/prom-label-proxy/0.log"
Jul 17 19:34:06.546713 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:06.546672    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8c076bd6-5c9d-4dc8-93de-e54c69cf8673/init-config-reloader/0.log"
Jul 17 19:34:06.592909 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:06.592870    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-xhz5l_39a38b78-ab33-411b-9a10-4cd932d6c535/cluster-monitoring-operator/0.log"
Jul 17 19:34:06.628889 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:06.628856    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-8pwvw_9306cc45-d78b-47ea-846a-3c0f881b8df8/kube-state-metrics/0.log"
Jul 17 19:34:06.647513 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:06.647433    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-8pwvw_9306cc45-d78b-47ea-846a-3c0f881b8df8/kube-rbac-proxy-main/0.log"
Jul 17 19:34:06.668064 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:06.668039    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-8pwvw_9306cc45-d78b-47ea-846a-3c0f881b8df8/kube-rbac-proxy-self/0.log"
Jul 17 19:34:06.853067 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:06.853021    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lzpxc_84cadd37-39b2-4702-95f3-b4cb570e8ba6/node-exporter/0.log"
Jul 17 19:34:06.875365 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:06.875334    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lzpxc_84cadd37-39b2-4702-95f3-b4cb570e8ba6/kube-rbac-proxy/0.log"
Jul 17 19:34:06.894931 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:06.894901    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lzpxc_84cadd37-39b2-4702-95f3-b4cb570e8ba6/init-textfile/0.log"
Jul 17 19:34:07.304529 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:07.304451    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75686874dd-fcc8n_bc08b863-6a47-4d6b-b163-fb3f4fb0bda4/telemeter-client/0.log"
Jul 17 19:34:07.335288 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:07.335262    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75686874dd-fcc8n_bc08b863-6a47-4d6b-b163-fb3f4fb0bda4/reload/0.log"
Jul 17 19:34:07.357235 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:07.357210    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-75686874dd-fcc8n_bc08b863-6a47-4d6b-b163-fb3f4fb0bda4/kube-rbac-proxy/0.log"
Jul 17 19:34:08.580621 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:08.580580    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"]
Jul 17 19:34:08.587248 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:08.587219    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"
Jul 17 19:34:08.590933 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:08.590219    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dtqk6\"/\"default-dockercfg-n4gpm\""
Jul 17 19:34:08.592596 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:08.592571    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"]
Jul 17 19:34:08.683052 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:08.683015    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb7e15f6-5bd9-4ce5-b573-762da85994f3-lib-modules\") pod \"perf-node-gather-daemonset-282tp\" (UID: \"cb7e15f6-5bd9-4ce5-b573-762da85994f3\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"
Jul 17 19:34:08.683362 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:08.683341    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb7e15f6-5bd9-4ce5-b573-762da85994f3-proc\") pod \"perf-node-gather-daemonset-282tp\" (UID: \"cb7e15f6-5bd9-4ce5-b573-762da85994f3\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"
Jul 17 19:34:08.683493 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:08.683478    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb7e15f6-5bd9-4ce5-b573-762da85994f3-sys\") pod \"perf-node-gather-daemonset-282tp\" (UID: \"cb7e15f6-5bd9-4ce5-b573-762da85994f3\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"
Jul 17 19:34:08.683604 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:08.683590    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76nzx\" (UniqueName: \"kubernetes.io/projected/cb7e15f6-5bd9-4ce5-b573-762da85994f3-kube-api-access-76nzx\") pod \"perf-node-gather-daemonset-282tp\" (UID: \"cb7e15f6-5bd9-4ce5-b573-762da85994f3\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"
Jul 17 19:34:08.683703 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:08.683692    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb7e15f6-5bd9-4ce5-b573-762da85994f3-podres\") pod \"perf-node-gather-daemonset-282tp\" (UID: \"cb7e15f6-5bd9-4ce5-b573-762da85994f3\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"
Jul 17 19:34:08.784881 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:08.784847    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb7e15f6-5bd9-4ce5-b573-762da85994f3-proc\") pod \"perf-node-gather-daemonset-282tp\" (UID: \"cb7e15f6-5bd9-4ce5-b573-762da85994f3\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"
Jul 17 19:34:08.785140 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:08.785110    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb7e15f6-5bd9-4ce5-b573-762da85994f3-sys\") pod \"perf-node-gather-daemonset-282tp\" (UID: \"cb7e15f6-5bd9-4ce5-b573-762da85994f3\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"
Jul 17 19:34:08.785382 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:08.785363    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76nzx\" (UniqueName: \"kubernetes.io/projected/cb7e15f6-5bd9-4ce5-b573-762da85994f3-kube-api-access-76nzx\") pod \"perf-node-gather-daemonset-282tp\" (UID: \"cb7e15f6-5bd9-4ce5-b573-762da85994f3\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"
Jul 17 19:34:08.785478 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:08.785408    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb7e15f6-5bd9-4ce5-b573-762da85994f3-podres\") pod \"perf-node-gather-daemonset-282tp\" (UID: \"cb7e15f6-5bd9-4ce5-b573-762da85994f3\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"
Jul 17 19:34:08.785560 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:08.785540    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb7e15f6-5bd9-4ce5-b573-762da85994f3-lib-modules\") pod \"perf-node-gather-daemonset-282tp\" (UID: \"cb7e15f6-5bd9-4ce5-b573-762da85994f3\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"
Jul 17 19:34:08.785754 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:08.785296    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb7e15f6-5bd9-4ce5-b573-762da85994f3-sys\") pod \"perf-node-gather-daemonset-282tp\" (UID: \"cb7e15f6-5bd9-4ce5-b573-762da85994f3\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"
Jul 17 19:34:08.785754 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:08.785725    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb7e15f6-5bd9-4ce5-b573-762da85994f3-podres\") pod \"perf-node-gather-daemonset-282tp\" (UID: \"cb7e15f6-5bd9-4ce5-b573-762da85994f3\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"
Jul 17 19:34:08.785754 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:08.785725    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb7e15f6-5bd9-4ce5-b573-762da85994f3-lib-modules\") pod \"perf-node-gather-daemonset-282tp\" (UID: \"cb7e15f6-5bd9-4ce5-b573-762da85994f3\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"
Jul 17 19:34:08.785947 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:08.785051    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb7e15f6-5bd9-4ce5-b573-762da85994f3-proc\") pod \"perf-node-gather-daemonset-282tp\" (UID: \"cb7e15f6-5bd9-4ce5-b573-762da85994f3\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"
Jul 17 19:34:08.795162 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:08.795140    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76nzx\" (UniqueName: \"kubernetes.io/projected/cb7e15f6-5bd9-4ce5-b573-762da85994f3-kube-api-access-76nzx\") pod \"perf-node-gather-daemonset-282tp\" (UID: \"cb7e15f6-5bd9-4ce5-b573-762da85994f3\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"
Jul 17 19:34:08.903264 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:08.903186    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"
Jul 17 19:34:09.095489 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:09.095255    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"]
Jul 17 19:34:09.099808 ip-10-0-138-51 kubenswrapper[2559]: W0717 19:34:09.099780    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcb7e15f6_5bd9_4ce5_b573_762da85994f3.slice/crio-a1bfda1b9a8aa30193b8cbcab68e2821435862e72e697fc5083bf8cd49126590 WatchSource:0}: Error finding container a1bfda1b9a8aa30193b8cbcab68e2821435862e72e697fc5083bf8cd49126590: Status 404 returned error can't find the container with id a1bfda1b9a8aa30193b8cbcab68e2821435862e72e697fc5083bf8cd49126590
Jul 17 19:34:09.331003 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:09.330967    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/1.log"
Jul 17 19:34:09.336545 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:09.336523    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-vv4vz_b5bd5031-5dc3-446b-88ea-83821a9f3faf/console-operator/2.log"
Jul 17 19:34:09.821297 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:09.821268    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-75755b7754-qmvld_a8fb8bab-54a3-4811-869c-122ff53c8e5f/console/0.log"
Jul 17 19:34:09.861438 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:09.861408    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-stqc8_7a1c2e51-24a8-49a2-ae10-75ce216f38f8/download-server/0.log"
Jul 17 19:34:10.009011 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:10.008975    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp" event={"ID":"cb7e15f6-5bd9-4ce5-b573-762da85994f3","Type":"ContainerStarted","Data":"8f7005ddefdd6ec8bf0365a915146b8848cb43ef7e10bfba62f7461fab8aa0fd"}
Jul 17 19:34:10.009212 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:10.009040    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp" event={"ID":"cb7e15f6-5bd9-4ce5-b573-762da85994f3","Type":"ContainerStarted","Data":"a1bfda1b9a8aa30193b8cbcab68e2821435862e72e697fc5083bf8cd49126590"}
Jul 17 19:34:10.009212 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:10.009180    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"
Jul 17 19:34:10.026259 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:10.026203    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp" podStartSLOduration=2.026184724 podStartE2EDuration="2.026184724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 19:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:34:10.023998435 +0000 UTC m=+3660.465507126" watchObservedRunningTime="2026-07-17 19:34:10.026184724 +0000 UTC m=+3660.467693413"
Jul 17 19:34:11.519875 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:11.519844    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7drcz_ab87a034-e286-4b47-8513-9fb3c0b53084/dns/0.log"
Jul 17 19:34:11.541147 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:11.541086    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7drcz_ab87a034-e286-4b47-8513-9fb3c0b53084/kube-rbac-proxy/0.log"
Jul 17 19:34:11.635235 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:11.635196    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-njrqr_3a1608b8-a53a-480d-965d-8ab7e3ec8f22/dns-node-resolver/0.log"
Jul 17 19:34:12.302490 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:12.302454    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ztfrm_d33d089a-3e2f-4edd-a459-0303cf90f1af/node-ca/0.log"
Jul 17 19:34:13.423964 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:13.423922    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864488b8fd-fxtq4_367eed95-dbb9-47d7-aa40-bc764b363849/kube-auth-proxy/0.log"
Jul 17 19:34:14.359359 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:14.359324    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lfvbt_0881b192-5909-49a5-ba16-65dbc6829189/serve-healthcheck-canary/0.log"
Jul 17 19:34:14.840031 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:14.840001    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-tmwrz_b92733f1-3721-44de-a7d9-c19deb63d6a4/insights-operator/0.log"
Jul 17 19:34:14.840958 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:14.840937    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-tmwrz_b92733f1-3721-44de-a7d9-c19deb63d6a4/insights-operator/1.log"
Jul 17 19:34:14.864085 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:14.864061    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bzs5r_a85d465e-b514-4d92-9869-898394cd3652/kube-rbac-proxy/0.log"
Jul 17 19:34:14.887465 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:14.887442    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bzs5r_a85d465e-b514-4d92-9869-898394cd3652/exporter/0.log"
Jul 17 19:34:14.911652 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:14.911624    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bzs5r_a85d465e-b514-4d92-9869-898394cd3652/extractor/0.log"
Jul 17 19:34:16.029612 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:16.029584    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-282tp"
Jul 17 19:34:17.650161 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:17.650132    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7c84488dd8-2nhq2_94530b49-743c-4bac-812f-897823b999b2/manager/0.log"
Jul 17 19:34:17.997798 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:17.997717    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57479f7bfb-xfmhk_101e4b60-dfa6-48e7-97e1-0de5c935dbef/manager/0.log"
Jul 17 19:34:18.016004 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:18.015949    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57479f7bfb-xfmhk_101e4b60-dfa6-48e7-97e1-0de5c935dbef/copy-manifests/0.log"
Jul 17 19:34:19.223762 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:19.223679    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864488b8fd-fxtq4_367eed95-dbb9-47d7-aa40-bc764b363849/kube-auth-proxy/0.log"
Jul 17 19:34:19.347879 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:19.347852    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-cbc4544f6-8wd95_194e96f5-01d6-45b5-b20d-a8423562d81b/manager/0.log"
Jul 17 19:34:20.709084 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:20.709048    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864488b8fd-fxtq4_367eed95-dbb9-47d7-aa40-bc764b363849/kube-auth-proxy/0.log"
Jul 17 19:34:24.498678 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:24.498646    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-cwbpb_03b5f935-6b84-4179-93f1-86e325c90c81/migrator/0.log"
Jul 17 19:34:24.517924 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:24.517893    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-cwbpb_03b5f935-6b84-4179-93f1-86e325c90c81/graceful-termination/0.log"
Jul 17 19:34:25.906474 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:25.906388    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-58rmc_57c3fb7d-dfc9-468c-a17a-37a625e1fb74/kube-multus/0.log"
Jul 17 19:34:26.081413 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:26.081389    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cnj2q_486589ec-aa0a-497e-88e6-79e52df71588/kube-multus-additional-cni-plugins/0.log"
Jul 17 19:34:26.101595 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:26.101572    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cnj2q_486589ec-aa0a-497e-88e6-79e52df71588/egress-router-binary-copy/0.log"
Jul 17 19:34:26.121199 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:26.121175    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cnj2q_486589ec-aa0a-497e-88e6-79e52df71588/cni-plugins/0.log"
Jul 17 19:34:26.143139 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:26.143103    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cnj2q_486589ec-aa0a-497e-88e6-79e52df71588/bond-cni-plugin/0.log"
Jul 17 19:34:26.164023 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:26.163925    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cnj2q_486589ec-aa0a-497e-88e6-79e52df71588/routeoverride-cni/0.log"
Jul 17 19:34:26.183853 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:26.183834    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cnj2q_486589ec-aa0a-497e-88e6-79e52df71588/whereabouts-cni-bincopy/0.log"
Jul 17 19:34:26.203704 ip-10-0-138-51 kubenswrapper[2559]: I0717 19:34:26.203681    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cnj2q_486589ec-aa0a-497e-88e6-79e52df71588/whereabouts-cni/0.log"