Jul 17 21:08:04.422269 ip-10-0-140-117 systemd[1]: Starting Kubernetes Kubelet...
Jul 17 21:08:04.762851 ip-10-0-140-117 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 21:08:04.762851 ip-10-0-140-117 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 17 21:08:04.762851 ip-10-0-140-117 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 21:08:04.762851 ip-10-0-140-117 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 17 21:08:04.762851 ip-10-0-140-117 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 21:08:04.762851 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.750247    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 17 21:08:04.762851 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755480    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:08:04.762851 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755495    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:08:04.762851 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755499    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:08:04.762851 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755502    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:08:04.762851 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755505    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:08:04.762851 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755508    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:08:04.762851 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755511    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:08:04.762851 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755513    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:08:04.781036 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755516    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:08:04.781036 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755519    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:08:04.781036 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755521    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:08:04.781036 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755524    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:08:04.781036 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755526    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:08:04.781036 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755529    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:08:04.781036 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755532    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:08:04.781036 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755535    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:08:04.781036 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755537    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:08:04.781036 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755540    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:08:04.781036 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755542    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:08:04.781036 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755545    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:08:04.781036 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755548    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:08:04.781036 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755550    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:08:04.781036 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755553    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:08:04.781036 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755556    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:08:04.781036 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755558    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:08:04.781036 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755561    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:08:04.781036 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755563    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:08:04.781036 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755566    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:08:04.783572 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755568    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:08:04.783572 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755572    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:08:04.783572 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755574    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:08:04.783572 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755576    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:08:04.783572 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755579    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:08:04.783572 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755581    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:08:04.783572 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755583    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:08:04.783572 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755586    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:08:04.783572 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755589    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:08:04.783572 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755592    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:08:04.783572 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755594    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:08:04.783572 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755596    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:08:04.783572 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755599    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:08:04.783572 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755601    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:08:04.783572 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755604    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:08:04.783572 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755606    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:08:04.783572 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755609    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:08:04.783572 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755611    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:08:04.783572 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755614    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:08:04.804713 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755617    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:08:04.804713 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755619    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:08:04.804713 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755622    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:08:04.804713 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755624    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:08:04.804713 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755627    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:08:04.804713 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755629    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:08:04.804713 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755632    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:08:04.804713 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755636    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:08:04.804713 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755639    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:08:04.804713 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755642    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:08:04.804713 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755645    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:08:04.804713 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755647    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:08:04.804713 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755650    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:08:04.804713 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755652    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:08:04.804713 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755655    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:08:04.804713 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755661    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:08:04.804713 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755664    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:08:04.804713 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755666    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:08:04.804713 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755669    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:08:04.807170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755672    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:08:04.807170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755675    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:08:04.807170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755678    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:08:04.807170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755680    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:08:04.807170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755683    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:08:04.807170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755685    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:08:04.807170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755688    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:08:04.807170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755690    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:08:04.807170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755692    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:08:04.807170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755695    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:08:04.807170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755697    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:08:04.807170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755700    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:08:04.807170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755703    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:08:04.807170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755708    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:08:04.807170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755711    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:08:04.807170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755714    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:08:04.807170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755717    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:08:04.807170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755720    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:08:04.807170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755722    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:08:04.807170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755725    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:08:04.808587 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755727    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:08:04.808587 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755730    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:08:04.808587 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755733    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:08:04.808587 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755735    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:08:04.808587 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755738    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:08:04.808587 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755740    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:08:04.808587 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755743    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:08:04.808587 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755745    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:08:04.808587 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755747    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:08:04.808587 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755755    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:08:04.808587 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755758    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:08:04.808587 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755761    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:08:04.808587 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.755764    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:08:04.808587 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757088    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:08:04.808587 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757096    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:08:04.808587 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757099    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:08:04.808587 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757103    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:08:04.808587 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757107    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:08:04.808587 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757110    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:08:04.811196 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757113    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:08:04.811196 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757116    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:08:04.811196 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757118    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:08:04.811196 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757121    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:08:04.811196 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757123    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:08:04.811196 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757126    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:08:04.811196 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757128    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:08:04.811196 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757131    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:08:04.811196 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757133    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:08:04.811196 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757136    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:08:04.811196 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757138    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:08:04.811196 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757141    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:08:04.811196 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757143    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:08:04.811196 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757146    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:08:04.811196 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757149    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:08:04.811196 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757151    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:08:04.811196 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757154    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:08:04.811196 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757156    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:08:04.811196 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757159    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:08:04.811196 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757162    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:08:04.813384 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757164    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:08:04.813384 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757167    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:08:04.813384 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757169    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:08:04.813384 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757172    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:08:04.813384 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757175    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:08:04.813384 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757177    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:08:04.813384 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757180    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:08:04.813384 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757182    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:08:04.813384 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757185    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:08:04.813384 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757187    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:08:04.813384 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757190    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:08:04.813384 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757192    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:08:04.813384 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757195    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:08:04.813384 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757197    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:08:04.813384 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757199    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:08:04.813384 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757202    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:08:04.813384 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757204    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:08:04.813384 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757206    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:08:04.813384 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757209    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:08:04.813384 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757211    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:08:04.816056 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757214    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:08:04.816056 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757216    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:08:04.816056 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757219    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:08:04.816056 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757221    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:08:04.816056 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757224    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:08:04.816056 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757226    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:08:04.816056 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757229    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:08:04.816056 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757231    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:08:04.816056 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757234    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:08:04.816056 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757236    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:08:04.816056 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757238    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:08:04.816056 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757241    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:08:04.816056 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757243    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:08:04.816056 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757246    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:08:04.816056 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757248    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:08:04.816056 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757250    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:08:04.816056 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757255    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:08:04.816056 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757257    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:08:04.816056 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757260    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:08:04.816056 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757262    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:08:04.818016 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757265    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:08:04.818016 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757267    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:08:04.818016 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757269    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:08:04.818016 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757272    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:08:04.818016 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757274    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:08:04.818016 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757276    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:08:04.818016 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757280    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:08:04.818016 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757283    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:08:04.818016 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757286    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:08:04.818016 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757288    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:08:04.818016 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757291    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:08:04.818016 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757293    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:08:04.818016 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757296    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:08:04.818016 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757298    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:08:04.818016 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757301    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:08:04.818016 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757304    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:08:04.818016 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757307    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:08:04.818016 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757309    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:08:04.818016 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757312    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:08:04.820604 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757316    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:08:04.820604 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757318    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:08:04.820604 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757321    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:08:04.820604 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757324    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:08:04.820604 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757326    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:08:04.820604 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757329    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:08:04.820604 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757331    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:08:04.820604 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757333    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:08:04.820604 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757336    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:08:04.820604 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757338    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:08:04.820604 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757341    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:08:04.820604 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757343    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:08:04.820604 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757347    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:08:04.820604 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757350    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:08:04.820604 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757785    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:08:04.820604 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757792    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:08:04.820604 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757795    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:08:04.820604 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757798    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:08:04.820604 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757801    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:08:04.820831 ip-10-0-140-117 systemd[1]: Started Kubernetes Kubelet.
Jul 17 21:08:04.835409 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757804    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:08:04.835409 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757807    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:08:04.835409 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757809    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:08:04.835409 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757812    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:08:04.835409 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757814    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:08:04.835409 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757817    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:08:04.835409 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757820    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:08:04.835409 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757822    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:08:04.835409 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757825    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:08:04.835409 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757828    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:08:04.835409 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757831    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:08:04.835409 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757833    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:08:04.835409 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757836    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:08:04.835409 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757838    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:08:04.835409 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757841    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:08:04.835409 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757844    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:08:04.835409 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757846    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:08:04.835409 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757848    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:08:04.835409 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757851    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:08:04.835409 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757853    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:08:04.836933 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757856    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:08:04.836933 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757858    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:08:04.836933 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757861    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:08:04.836933 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757864    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:08:04.836933 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757868    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:08:04.836933 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757871    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:08:04.836933 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757874    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:08:04.836933 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757876    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:08:04.836933 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757878    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:08:04.836933 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757881    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:08:04.836933 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757884    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:08:04.836933 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757886    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:08:04.836933 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757888    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:08:04.836933 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757891    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:08:04.836933 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757895    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:08:04.836933 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757897    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:08:04.836933 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757900    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:08:04.836933 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757902    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:08:04.836933 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757905    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:08:04.851290 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757907    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:08:04.851290 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757909    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:08:04.851290 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757912    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:08:04.851290 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757914    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:08:04.851290 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757917    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:08:04.851290 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757920    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:08:04.851290 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757922    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:08:04.851290 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757926    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:08:04.851290 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757929    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:08:04.851290 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757932    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:08:04.851290 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757934    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:08:04.851290 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757937    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:08:04.851290 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757940    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:08:04.851290 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757942    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:08:04.851290 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757947    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:08:04.851290 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757950    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:08:04.851290 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757954    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:08:04.851290 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757959    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:08:04.851290 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757977    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:08:04.854457 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757981    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:08:04.854457 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757984    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:08:04.854457 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757987    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:08:04.854457 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757990    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:08:04.854457 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757993    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:08:04.854457 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757996    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:08:04.854457 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.757999    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:08:04.854457 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758002    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:08:04.854457 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758005    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:08:04.854457 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758008    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:08:04.854457 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758010    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:08:04.854457 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758013    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:08:04.854457 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758016    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:08:04.854457 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758019    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:08:04.854457 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758021    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:08:04.854457 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758024    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:08:04.854457 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758026    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:08:04.854457 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758029    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:08:04.854457 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758031    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:08:04.854457 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758034    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758036    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758039    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758041    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758043    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758046    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758048    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758050    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758053    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758055    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758058    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758062    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758065    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758067    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758070    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758072    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758075    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758141    2569 flags.go:64] FLAG: --address="0.0.0.0"
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758151    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758160    2569 flags.go:64] FLAG: --anonymous-auth="true"
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758166    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 17 21:08:04.867837 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758174    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758178    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758182    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758198    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758201    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758204    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758208    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758211    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758214    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758217    2569 flags.go:64] FLAG: --cgroup-root=""
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758220    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758223    2569 flags.go:64] FLAG: --client-ca-file=""
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758243    2569 flags.go:64] FLAG: --cloud-provider="external"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758246    2569 flags.go:64] FLAG: --cluster-dns="[]"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758250    2569 flags.go:64] FLAG: --cluster-domain=""
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758253    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758256    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758259    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758262    2569 flags.go:64] FLAG: --container-log-max-files="5"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758266    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758269    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758272    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758277    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758280    2569 flags.go:64] FLAG: --contention-profiling="false"
Jul 17 21:08:04.911166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758284    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758287    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758291    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758293    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758297    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758300    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758303    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758306    2569 flags.go:64] FLAG: --enable-load-reader="false"
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758311    2569 flags.go:64] FLAG: --enable-server="true"
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758314    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758318    2569 flags.go:64] FLAG: --event-burst="100"
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758321    2569 flags.go:64] FLAG: --event-qps="50"
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758324    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758327    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758331    2569 flags.go:64] FLAG: --eviction-hard=""
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758334    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758337    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758340    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758343    2569 flags.go:64] FLAG: --eviction-soft=""
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758347    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758349    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758352    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758355    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758358    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758361    2569 flags.go:64] FLAG: --fail-swap-on="true"
Jul 17 21:08:05.040007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758364    2569 flags.go:64] FLAG: --feature-gates=""
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758368    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758371    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758374    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758377    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758381    2569 flags.go:64] FLAG: --healthz-port="10248"
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758384    2569 flags.go:64] FLAG: --help="false"
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758387    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-140-117.ec2.internal"
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758390    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758393    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758396    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758400    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758403    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758406    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758409    2569 flags.go:64] FLAG: --image-service-endpoint=""
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758412    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758415    2569 flags.go:64] FLAG: --kube-api-burst="100"
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758418    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758421    2569 flags.go:64] FLAG: --kube-api-qps="50"
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758424    2569 flags.go:64] FLAG: --kube-reserved=""
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758427    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758429    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758432    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758435    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 17 21:08:05.041005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758438    2569 flags.go:64] FLAG: --lock-file=""
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758441    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758444    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758447    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758452    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758455    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758458    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758460    2569 flags.go:64] FLAG: --logging-format="text"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758463    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758467    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758469    2569 flags.go:64] FLAG: --manifest-url=""
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758472    2569 flags.go:64] FLAG: --manifest-url-header=""
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758476    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758480    2569 flags.go:64] FLAG: --max-open-files="1000000"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758484    2569 flags.go:64] FLAG: --max-pods="110"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758487    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758490    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758493    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758496    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758499    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758502    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758505    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758511    2569 flags.go:64] FLAG: --node-status-max-images="50"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758516    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758519    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 17 21:08:05.042271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758522    2569 flags.go:64] FLAG: --pod-cidr=""
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758525    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758530    2569 flags.go:64] FLAG: --pod-manifest-path=""
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758533    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758537    2569 flags.go:64] FLAG: --pods-per-core="0"
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758540    2569 flags.go:64] FLAG: --port="10250"
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758543    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758546    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06c931f73db65fd48"
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758549    2569 flags.go:64] FLAG: --qos-reserved=""
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758552    2569 flags.go:64] FLAG: --read-only-port="10255"
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758555    2569 flags.go:64] FLAG: --register-node="true"
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758558    2569 flags.go:64] FLAG: --register-with-taints=""
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758562    2569 flags.go:64] FLAG: --registry-burst="10"
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758565    2569 flags.go:64] FLAG: --registry-qps="5"
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758568    2569 flags.go:64] FLAG: --reserved-cpus=""
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758570    2569 flags.go:64] FLAG: --reserved-memory=""
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758574    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758577    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758580    2569 flags.go:64] FLAG: --rotate-certificates="false"
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758583    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758586    2569 flags.go:64] FLAG: --runonce="false"
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758589    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758593    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758596    2569 flags.go:64] FLAG: --seccomp-default="false"
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758598    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 17 21:08:05.043311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758601    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758604    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758607    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758610    2569 flags.go:64] FLAG: --storage-driver-password="root"
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758613    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758618    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758620    2569 flags.go:64] FLAG: --storage-driver-user="root"
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758623    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758627    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758629    2569 flags.go:64] FLAG: --system-cgroups=""
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758632    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758637    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758640    2569 flags.go:64] FLAG: --tls-cert-file=""
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758643    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758646    2569 flags.go:64] FLAG: --tls-min-version=""
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758649    2569 flags.go:64] FLAG: --tls-private-key-file=""
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758652    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758655    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758658    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758661    2569 flags.go:64] FLAG: --v="2"
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758670    2569 flags.go:64] FLAG: --version="false"
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758674    2569 flags.go:64] FLAG: --vmodule=""
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758677    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.758681    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758815    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:08:05.044585 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758819    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:08:05.045427 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758822    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:08:05.045427 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758825    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:08:05.045427 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758830    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:08:05.045427 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758833    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:08:05.045427 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758835    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:08:05.045427 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758838    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:08:05.045427 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758841    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:08:05.045427 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758845    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:08:05.045427 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758848    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:08:05.045427 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758850    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:08:05.045427 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758853    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:08:05.045427 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758857    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:08:05.045427 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758860    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:08:05.045427 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758863    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:08:05.045427 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758865    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:08:05.045427 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758868    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:08:05.045427 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758871    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:08:05.045427 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758873    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:08:05.045427 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758876    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:08:05.046327 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758878    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:08:05.046327 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758882    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:08:05.046327 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758885    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:08:05.046327 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758888    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:08:05.046327 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758890    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:08:05.046327 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758893    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:08:05.046327 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758896    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:08:05.046327 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758899    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:08:05.046327 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758902    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:08:05.046327 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758904    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:08:05.046327 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758907    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:08:05.046327 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758910    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:08:05.046327 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758913    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:08:05.046327 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758915    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:08:05.046327 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758918    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:08:05.046327 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758921    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:08:05.046327 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758924    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:08:05.046327 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758927    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:08:05.046327 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758929    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:08:05.047766 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758932    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:08:05.047766 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758935    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:08:05.047766 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758937    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:08:05.047766 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758940    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:08:05.047766 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758942    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:08:05.047766 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758946    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:08:05.047766 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758948    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:08:05.047766 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758951    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:08:05.047766 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758953    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:08:05.047766 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758956    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:08:05.047766 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758958    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:08:05.047766 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758961    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:08:05.047766 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758978    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:08:05.047766 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758981    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:08:05.047766 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758984    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:08:05.047766 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758987    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:08:05.047766 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758991    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:08:05.047766 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758994    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:08:05.047766 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.758997    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:08:05.048346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759000    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:08:05.048346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759003    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:08:05.048346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759005    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:08:05.048346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759008    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:08:05.048346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759010    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:08:05.048346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759013    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:08:05.048346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759015    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:08:05.048346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759018    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:08:05.048346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759020    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:08:05.048346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759024    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:08:05.048346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759026    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:08:05.048346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759029    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:08:05.048346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759032    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:08:05.048346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759034    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:08:05.048346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759037    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:08:05.048346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759039    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:08:05.048346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759043    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:08:05.048346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759045    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:08:05.048346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759050    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:08:05.048346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759052    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:08:05.048989 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759055    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:08:05.048989 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759069    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:08:05.048989 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759073    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:08:05.048989 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759076    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:08:05.048989 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759080    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:08:05.048989 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759083    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:08:05.048989 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759085    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:08:05.048989 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759088    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:08:05.048989 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759091    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:08:05.048989 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759093    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:08:05.048989 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759096    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:08:05.048989 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759099    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:08:05.048989 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759101    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:08:05.048989 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759104    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:08:05.048989 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759107    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:08:05.048989 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759110    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:08:05.048989 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759114    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:08:05.048989 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759116    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:08:05.048989 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759119    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:08:05.048989 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.759121    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:08:05.049601 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.759618    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 21:08:05.049601 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.770397    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 17 21:08:05.049601 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.770414    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 17 21:08:05.049601 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.770434    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 17 21:08:05.049601 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.770439    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 17 21:08:05.049601 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770500    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:08:05.049601 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770505    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:08:05.049601 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770509    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:08:05.049601 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770512    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:08:05.049601 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770515    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:08:05.049601 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770518    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:08:05.049601 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770521    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:08:05.049601 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770525    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:08:05.049601 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770529    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:08:05.050446 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770533    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:08:05.050446 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770536    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:08:05.050446 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770539    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:08:05.050446 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770542    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:08:05.050446 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770545    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:08:05.050446 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770548    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:08:05.050446 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770551    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:08:05.050446 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770554    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:08:05.050446 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770558    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:08:05.050446 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770561    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:08:05.050446 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770564    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:08:05.050446 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770567    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:08:05.050446 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770570    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:08:05.050446 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770573    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:08:05.050446 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770576    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:08:05.050446 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770579    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:08:05.050446 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770581    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:08:05.050446 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770584    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:08:05.050446 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770586    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:08:05.050446 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770589    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:08:05.051072 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770592    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:08:05.051072 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770595    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:08:05.051072 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770598    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:08:05.051072 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770601    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:08:05.051072 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770604    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:08:05.051072 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770606    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:08:05.051072 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770609    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:08:05.051072 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770612    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:08:05.051072 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770614    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:08:05.051072 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770617    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:08:05.051072 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770619    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:08:05.051072 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770622    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:08:05.051072 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770624    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:08:05.051072 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770626    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:08:05.051072 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770629    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:08:05.051072 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770632    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:08:05.051072 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770634    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:08:05.051072 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770643    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:08:05.051072 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770646    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:08:05.051072 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770650    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:08:05.052305 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770653    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:08:05.052305 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770656    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:08:05.052305 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770659    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:08:05.052305 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770661    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:08:05.052305 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770664    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:08:05.052305 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770666    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:08:05.052305 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770669    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:08:05.052305 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770672    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:08:05.052305 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770674    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:08:05.052305 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770677    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:08:05.052305 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770679    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:08:05.052305 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770682    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:08:05.052305 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770685    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:08:05.052305 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770687    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:08:05.052305 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770690    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:08:05.052305 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770693    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:08:05.052305 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770695    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:08:05.052305 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770698    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:08:05.052305 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770700    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:08:05.052828 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770705    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:08:05.052828 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770708    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:08:05.052828 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770711    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:08:05.052828 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770714    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:08:05.052828 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770717    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:08:05.052828 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770720    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:08:05.052828 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770722    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:08:05.052828 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770725    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:08:05.052828 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770728    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:08:05.052828 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770730    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:08:05.052828 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770733    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:08:05.052828 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770741    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:08:05.052828 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770744    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:08:05.052828 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770747    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:08:05.052828 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770750    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:08:05.052828 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770752    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:08:05.052828 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770755    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:08:05.052828 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770758    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:08:05.052828 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770760    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:08:05.053405 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770763    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:08:05.053405 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770765    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:08:05.053405 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770768    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:08:05.053405 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770770    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:08:05.053405 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770773    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:08:05.053405 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770775    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:08:05.053405 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770778    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:08:05.053405 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770780    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:08:05.053405 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770783    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:08:05.053405 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770786    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:08:05.053405 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770788    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:08:05.053405 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770791    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:08:05.053405 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.770796    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 21:08:05.053405 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770911    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:08:05.053405 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770916    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:08:05.053845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770919    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:08:05.053845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770922    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:08:05.053845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770925    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:08:05.053845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770928    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:08:05.053845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770931    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:08:05.053845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770934    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:08:05.053845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770937    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:08:05.053845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770939    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:08:05.053845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770942    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:08:05.053845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770945    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:08:05.053845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770948    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:08:05.053845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770951    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:08:05.053845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770956    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:08:05.053845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770958    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:08:05.053845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770975    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:08:05.053845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770980    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:08:05.053845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770983    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:08:05.053845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770986    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:08:05.053845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770989    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:08:05.053845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770991    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:08:05.054677 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770994    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:08:05.054677 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.770997    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:08:05.054677 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771000    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:08:05.054677 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771002    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:08:05.054677 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771005    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:08:05.054677 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771008    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:08:05.054677 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771011    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:08:05.054677 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771013    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:08:05.054677 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771016    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:08:05.054677 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771019    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:08:05.054677 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771022    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:08:05.054677 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771024    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:08:05.054677 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771027    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:08:05.054677 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771029    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:08:05.054677 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771031    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:08:05.054677 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771034    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:08:05.054677 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771036    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:08:05.054677 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771039    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:08:05.054677 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771042    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:08:05.054677 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771045    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:08:05.055298 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771048    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:08:05.055298 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771050    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:08:05.055298 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771053    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:08:05.055298 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771056    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:08:05.055298 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771060    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:08:05.055298 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771063    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:08:05.055298 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771066    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:08:05.055298 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771068    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:08:05.055298 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771071    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:08:05.055298 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771074    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:08:05.055298 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771078    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:08:05.055298 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771081    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:08:05.055298 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771084    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:08:05.055298 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771087    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:08:05.055298 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771089    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:08:05.055298 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771092    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:08:05.055298 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771094    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:08:05.055298 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771097    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:08:05.055298 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771100    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:08:05.055298 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771103    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:08:05.055935 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771106    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:08:05.055935 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771109    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:08:05.055935 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771112    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:08:05.055935 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771115    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:08:05.055935 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771118    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:08:05.055935 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771121    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:08:05.055935 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771124    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:08:05.055935 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771127    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:08:05.055935 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771130    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:08:05.055935 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771132    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:08:05.055935 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771135    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:08:05.055935 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771138    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:08:05.055935 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771141    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:08:05.055935 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771145    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:08:05.055935 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771147    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:08:05.055935 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771150    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:08:05.055935 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771155    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:08:05.055935 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771157    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:08:05.055935 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771160    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:08:05.056470 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771163    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:08:05.056470 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771165    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:08:05.056470 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771168    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:08:05.056470 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771171    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:08:05.056470 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771173    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:08:05.056470 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771175    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:08:05.056470 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771178    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:08:05.056470 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771181    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:08:05.056470 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771183    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:08:05.056470 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771185    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:08:05.056470 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771188    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:08:05.056470 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771190    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:08:05.056470 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771193    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:08:05.056470 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771196    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:08:05.056470 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771199    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:08:05.056470 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771203    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:08:05.056470 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771205    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:08:05.056470 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:04.771208    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:08:05.059537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.771213    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 21:08:05.059537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.771320    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 17 21:08:05.059537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.773898    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 17 21:08:05.059537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.775087    2569 server.go:1019] "Starting client certificate rotation"
Jul 17 21:08:05.059537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.775184    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 21:08:05.059537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.775213    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 21:08:05.059537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.777134    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 21:08:05.059537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.778288    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 21:08:05.059537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.788233    2569 log.go:25] "Validated CRI v1 runtime API"
Jul 17 21:08:05.059537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.790757    2569 log.go:25] "Validated CRI v1 image API"
Jul 17 21:08:05.059537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.791587    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 17 21:08:05.059537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.794294    2569 fs.go:135] Filesystem UUIDs: map[5bdf079a-e876-4b0c-b972-560a359cf12b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a2d9b27c-5710-4d65-9e81-4bc650dde3fc:/dev/nvme0n1p3]
Jul 17 21:08:05.059983 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.794310    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 17 21:08:05.060078 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.800047    2569 manager.go:216] Machine: {Timestamp:2026-07-17 21:08:04.79955683 +0000 UTC m=+0.280559013 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3195399 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26a9436706a75c3eb557553dbd07dc SystemUUID:ec26a943-6706-a75c-3eb5-57553dbd07dc BootID:6a863217-77f0-43bc-8d49-2c59eaaa6188 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ee:ef:6c:3f:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ee:ef:6c:3f:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:49:94:f9:1e:dc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 17 21:08:05.060078 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.800147    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 17 21:08:05.060078 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.800222    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 17 21:08:05.060078 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.801991    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 17 21:08:05.060078 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.802017    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-117.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 17 21:08:05.060078 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.802520    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 21:08:05.060078 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.803043    2569 topology_manager.go:138] "Creating topology manager with none policy"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.803055    2569 container_manager_linux.go:309] "Creating device plugin manager"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.803063    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.803602    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.804773    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.809008    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.809122    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.809147    2569 kubelet.go:486] "Attempting to sync node with API server"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.809156    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.809167    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.809176    2569 kubelet.go:398] "Adding apiserver pod source"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.809184    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.809456    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.809469    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.812050    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.813957    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.814004    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.814824    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.814841    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.814850    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.814859    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.814871    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.814880    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.814890    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.814898    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.814907    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.814916    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.814926    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.814933    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.815625    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.815636    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:04.819741    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:04.819741    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-117.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.819887    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-117.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.820006    2569 server.go:1268] "Started kubelet"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.820629    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.820703    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.821174    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.822326    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 17 21:08:05.061767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.822792    2569 server.go:310] "Adding debug handlers to kubelet server"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:04.824725    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-117.ec2.internal.18c32fd8a06ffbf1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-117.ec2.internal,UID:ip-10-0-140-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-117.ec2.internal,},FirstTimestamp:2026-07-17 21:08:04.819983345 +0000 UTC m=+0.300985529,LastTimestamp:2026-07-17 21:08:04.819983345 +0000 UTC m=+0.300985529,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-117.ec2.internal,}"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:04.826642    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.828084    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.828097    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:04.828169    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-117.ec2.internal\" not found"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.828569    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.828580    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.828603    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.828671    2569 reconstruct.go:97] "Volume reconstruction finished"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.828680    2569 reconciler.go:29] "Reconciler: start to sync state"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.828841    2569 factory.go:55] Registering systemd factory
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.828855    2569 factory.go:223] Registration of the systemd container factory successfully
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.829159    2569 factory.go:153] Registering CRI-O factory
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.829174    2569 factory.go:223] Registration of the crio container factory successfully
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.829233    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.829256    2569 factory.go:103] Registering Raw factory
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.829268    2569 manager.go:1196] Started watching for new ooms in manager
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.829669    2569 manager.go:349] Starting recovery of all containers
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:04.842868    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:04.842980    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-117.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.843123    2569 manager.go:354] Recovery completed
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.847330    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.847859    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7bjk4"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.849223    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.849251    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.849264    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.849753    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.849760    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.849773    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.849814    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.851277    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:08:05.064435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.851308    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.851323    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.851960    2569 policy_none.go:49] "None policy: Start"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.851994    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.852009    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.854285    2569 policy_none.go:47] "Start"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.855506    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7bjk4"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:04.858737    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-117.ec2.internal.18c32fd8a22e520f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-117.ec2.internal,UID:ip-10-0-140-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-117.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-117.ec2.internal,},FirstTimestamp:2026-07-17 21:08:04.849234447 +0000 UTC m=+0.330236629,LastTimestamp:2026-07-17 21:08:04.849234447 +0000 UTC m=+0.330236629,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-117.ec2.internal,}"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.906544    2569 manager.go:337] "Starting Device Plugin manager"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:04.906573    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.906583    2569 server.go:87] "Starting device plugin registration server"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.906817    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.906828    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.906949    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.907052    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.907061    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:04.907511    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:04.907559    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-117.ec2.internal\" not found"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.927010    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.928199    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.928232    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.928252    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:04.928284    2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:04.931390    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.007634    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.008610    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.008648    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.008663    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.008694    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-117.ec2.internal"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.018706    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-117.ec2.internal"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:05.018726    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-117.ec2.internal\": node \"ip-10-0-140-117.ec2.internal\" not found"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.028657    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-117.ec2.internal"]
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.028722    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:08:05.066203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.029948    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:08:05.069175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.029988    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:08:05.069175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.030001    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:08:05.069175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.032384    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:08:05.069175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.032481    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal"
Jul 17 21:08:05.069175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.032506    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:08:05.069175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.033063    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:08:05.069175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.033090    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:08:05.069175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.033102    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:08:05.069175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.033110    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:08:05.069175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.033129    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:08:05.069175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.033138    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:08:05.069175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.035337    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-117.ec2.internal"
Jul 17 21:08:05.069175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.035364    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:08:05.069175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.036185    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:08:05.069175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.036204    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:08:05.069175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.036213    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:08:05.069175 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:05.047928    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-117.ec2.internal\" not found"
Jul 17 21:08:05.205922 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:05.079161    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-117.ec2.internal\" not found" node="ip-10-0-140-117.ec2.internal"
Jul 17 21:08:05.205922 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:05.082711    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-117.ec2.internal\" not found" node="ip-10-0-140-117.ec2.internal"
Jul 17 21:08:05.205922 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:05.148482    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-117.ec2.internal\" not found"
Jul 17 21:08:05.313238 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.229386    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dd2d9339de65db528a31e497b009a984-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal\" (UID: \"dd2d9339de65db528a31e497b009a984\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal"
Jul 17 21:08:05.313238 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.229465    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd2d9339de65db528a31e497b009a984-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal\" (UID: \"dd2d9339de65db528a31e497b009a984\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal"
Jul 17 21:08:05.313238 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.229486    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/533b658e3527087c58345fec12958890-config\") pod \"kube-apiserver-proxy-ip-10-0-140-117.ec2.internal\" (UID: \"533b658e3527087c58345fec12958890\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-117.ec2.internal"
Jul 17 21:08:05.313238 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:05.249424    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-117.ec2.internal\" not found"
Jul 17 21:08:05.439243 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.330209    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dd2d9339de65db528a31e497b009a984-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal\" (UID: \"dd2d9339de65db528a31e497b009a984\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal"
Jul 17 21:08:05.439243 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.330282    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dd2d9339de65db528a31e497b009a984-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal\" (UID: \"dd2d9339de65db528a31e497b009a984\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal"
Jul 17 21:08:05.439243 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.330680    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd2d9339de65db528a31e497b009a984-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal\" (UID: \"dd2d9339de65db528a31e497b009a984\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal"
Jul 17 21:08:05.439243 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.330691    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd2d9339de65db528a31e497b009a984-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal\" (UID: \"dd2d9339de65db528a31e497b009a984\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal"
Jul 17 21:08:05.439243 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.331002    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/533b658e3527087c58345fec12958890-config\") pod \"kube-apiserver-proxy-ip-10-0-140-117.ec2.internal\" (UID: \"533b658e3527087c58345fec12958890\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-117.ec2.internal"
Jul 17 21:08:05.439243 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.331113    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/533b658e3527087c58345fec12958890-config\") pod \"kube-apiserver-proxy-ip-10-0-140-117.ec2.internal\" (UID: \"533b658e3527087c58345fec12958890\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-117.ec2.internal"
Jul 17 21:08:05.439243 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:05.350283    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-117.ec2.internal\" not found"
Jul 17 21:08:05.439243 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.381453    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal"
Jul 17 21:08:05.439243 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.386248    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-117.ec2.internal"
Jul 17 21:08:05.545216 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:05.451203    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-117.ec2.internal\" not found"
Jul 17 21:08:05.621399 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:05.551893    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-117.ec2.internal\" not found"
Jul 17 21:08:05.729639 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:05.652370    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-117.ec2.internal\" not found"
Jul 17 21:08:05.830411 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:05.752918    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-117.ec2.internal\" not found"
Jul 17 21:08:05.830411 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.775068    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 17 21:08:05.830411 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.775209    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 21:08:05.834314 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.833716    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:08:05.836607 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.835083    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:08:05.861645 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.861611    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 21:03:06 +0000 UTC" deadline="2028-01-15 14:49:53.635286977 +0000 UTC"
Jul 17 21:08:05.861645 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.861640    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13121h41m47.77365001s"
Jul 17 21:08:05.928976 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.928935    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal"
Jul 17 21:08:05.942058 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.942036    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 21:08:05.942922 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.942906    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-117.ec2.internal"
Jul 17 21:08:05.951159 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:05.951140    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 21:08:07.734391 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:07.734357    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2d9339de65db528a31e497b009a984.slice/crio-54ab11e8eff9f1e58c26945b7b958da189174d7b672398d7ae2f5c6f41dca9d8 WatchSource:0}: Error finding container 54ab11e8eff9f1e58c26945b7b958da189174d7b672398d7ae2f5c6f41dca9d8: Status 404 returned error can't find the container with id 54ab11e8eff9f1e58c26945b7b958da189174d7b672398d7ae2f5c6f41dca9d8
Jul 17 21:08:07.738141 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:07.738126    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 21:08:07.816465 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:07.816438    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:08:08.314049 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:08.314013    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533b658e3527087c58345fec12958890.slice/crio-976632af211c042a60e7d3d759f2f824871916b07c0735d892feeb34a0a1ce70 WatchSource:0}: Error finding container 976632af211c042a60e7d3d759f2f824871916b07c0735d892feeb34a0a1ce70: Status 404 returned error can't find the container with id 976632af211c042a60e7d3d759f2f824871916b07c0735d892feeb34a0a1ce70
Jul 17 21:08:08.532960 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.532929    2569 apiserver.go:52] "Watching apiserver"
Jul 17 21:08:08.538195 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.538167    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 17 21:08:08.538630 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.538600    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-hnhk2","openshift-network-operator/iptables-alerter-2zqxd","openshift-ovn-kubernetes/ovnkube-node-4ccjx","kube-system/konnectivity-agent-7hlrp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg","openshift-cluster-node-tuning-operator/tuned-5lckv","openshift-dns/node-resolver-6fvtz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal","openshift-multus/multus-additional-cni-plugins-hxjhv","openshift-network-diagnostics/network-check-target-kphjs","kube-system/global-pull-secret-syncer-6bxwh","kube-system/kube-apiserver-proxy-ip-10-0-140-117.ec2.internal","openshift-image-registry/node-ca-qn5pr","openshift-multus/multus-djkwb"]
Jul 17 21:08:08.541395 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.541355    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6fvtz"
Jul 17 21:08:08.543332 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.543198    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tppfj\""
Jul 17 21:08:08.543332 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.543212    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 17 21:08:08.543332 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.543212    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 17 21:08:08.543813 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.543794    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qn5pr"
Jul 17 21:08:08.546063 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.545885    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 17 21:08:08.546063 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.545969    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 17 21:08:08.546210 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.546102    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.546309 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.546288    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 17 21:08:08.546621 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.546603    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7wgs9\""
Jul 17 21:08:08.548300 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.547995    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 17 21:08:08.548300 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.548030    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 17 21:08:08.548300 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.548173    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 17 21:08:08.548300 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.548184    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qpx9f\""
Jul 17 21:08:08.548300 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.548259    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 17 21:08:08.548608 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.548469    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7hlrp"
Jul 17 21:08:08.550100 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.550079    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bnb28\""
Jul 17 21:08:08.550173 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.550132    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 17 21:08:08.550173 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.550139    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 17 21:08:08.550290 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.550247    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 17 21:08:08.550612 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.550596    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:08.550690 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:08.550660    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kphjs" podUID="3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca"
Jul 17 21:08:08.553016 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.553000    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2zqxd"
Jul 17 21:08:08.554808 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.554787    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:08:08.554908 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.554869    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 17 21:08:08.555438 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.555107    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 17 21:08:08.555438 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.554794    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-przjx\""
Jul 17 21:08:08.555649 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.555606    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.558247 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.558037    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 17 21:08:08.558247 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.558201    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b86v4\""
Jul 17 21:08:08.558385 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.558298    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 17 21:08:08.558385 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.558335    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 17 21:08:08.558690 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.558672    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.558766 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.558722    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-host-run-multus-certs\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.558766 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.558755    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-host-var-lib-cni-bin\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.558867 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.558781    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-system-cni-dir\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.558867 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.558810    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tkkc\" (UniqueName: \"kubernetes.io/projected/867f0a7f-0397-4df5-8832-c6ee935d00a3-kube-api-access-2tkkc\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.558867 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.558842    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5c629464-bbea-40de-a8d0-8d31fa115b25-agent-certs\") pod \"konnectivity-agent-7hlrp\" (UID: \"5c629464-bbea-40de-a8d0-8d31fa115b25\") " pod="kube-system/konnectivity-agent-7hlrp"
Jul 17 21:08:08.559005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.558876    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-host-run-k8s-cni-cncf-io\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.559005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.558907    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-host-var-lib-cni-multus\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.559005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.558931    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs4xg\" (UniqueName: \"kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg\") pod \"network-check-target-kphjs\" (UID: \"3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca\") " pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:08.559005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.558961    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-multus-cni-dir\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.559005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.558998    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-multus-socket-dir-parent\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.560109 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.560089    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9d9e563-636a-4912-8557-a9d71096efb8-host\") pod \"node-ca-qn5pr\" (UID: \"a9d9e563-636a-4912-8557-a9d71096efb8\") " pod="openshift-image-registry/node-ca-qn5pr"
Jul 17 21:08:08.560183 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.560119    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9d9e563-636a-4912-8557-a9d71096efb8-serviceca\") pod \"node-ca-qn5pr\" (UID: \"a9d9e563-636a-4912-8557-a9d71096efb8\") " pod="openshift-image-registry/node-ca-qn5pr"
Jul 17 21:08:08.560183 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.560161    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-cnibin\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.560293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.560181    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/867f0a7f-0397-4df5-8832-c6ee935d00a3-cni-binary-copy\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.560293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.560202    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-hostroot\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.560293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.560224    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/867f0a7f-0397-4df5-8832-c6ee935d00a3-multus-daemon-config\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.560293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.560280    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-etc-kubernetes\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.560509 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.560301    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5c629464-bbea-40de-a8d0-8d31fa115b25-konnectivity-ca\") pod \"konnectivity-agent-7hlrp\" (UID: \"5c629464-bbea-40de-a8d0-8d31fa115b25\") " pod="kube-system/konnectivity-agent-7hlrp"
Jul 17 21:08:08.560509 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.560340    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98470465-ed0c-4896-8871-e8d9354e3dfa-tmp-dir\") pod \"node-resolver-6fvtz\" (UID: \"98470465-ed0c-4896-8871-e8d9354e3dfa\") " pod="openshift-dns/node-resolver-6fvtz"
Jul 17 21:08:08.560509 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.560360    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-host-run-netns\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.560509 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.560370    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dnvbk\""
Jul 17 21:08:08.560509 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.560387    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-host-var-lib-kubelet\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.560509 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.560475    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/98470465-ed0c-4896-8871-e8d9354e3dfa-hosts-file\") pod \"node-resolver-6fvtz\" (UID: \"98470465-ed0c-4896-8871-e8d9354e3dfa\") " pod="openshift-dns/node-resolver-6fvtz"
Jul 17 21:08:08.560509 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.560500    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6g7s\" (UniqueName: \"kubernetes.io/projected/98470465-ed0c-4896-8871-e8d9354e3dfa-kube-api-access-g6g7s\") pod \"node-resolver-6fvtz\" (UID: \"98470465-ed0c-4896-8871-e8d9354e3dfa\") " pod="openshift-dns/node-resolver-6fvtz"
Jul 17 21:08:08.560821 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.560524    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzf85\" (UniqueName: \"kubernetes.io/projected/a9d9e563-636a-4912-8557-a9d71096efb8-kube-api-access-hzf85\") pod \"node-ca-qn5pr\" (UID: \"a9d9e563-636a-4912-8557-a9d71096efb8\") " pod="openshift-image-registry/node-ca-qn5pr"
Jul 17 21:08:08.560821 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.560565    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-os-release\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.560821 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.560603    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 17 21:08:08.560821 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.560631    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-multus-conf-dir\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.560821 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.560645    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 17 21:08:08.561203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.561188    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.562512 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.562393    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 21:08:08.563062 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.562881    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 17 21:08:08.563062 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.562947    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mcghg\""
Jul 17 21:08:08.563062 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.562977    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:08:08.563506 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.563487    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:08.563611 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:08.563592    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhk2" podUID="89bdc455-1935-456b-bc58-288a0ed415a0"
Jul 17 21:08:08.565963 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.565945    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:08.566038 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:08.566014    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bxwh" podUID="710a2804-cc1e-49f5-8842-d392bb9aff26"
Jul 17 21:08:08.568298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.568277    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.570331 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.570181    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 17 21:08:08.570331 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.570193    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 17 21:08:08.570331 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.570185    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-st79t\""
Jul 17 21:08:08.570792 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.570624    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 17 21:08:08.570792 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.570697    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 17 21:08:08.570792 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.570764    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 17 21:08:08.570943 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.570904    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 17 21:08:08.588622 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.588604    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cmsvx"
Jul 17 21:08:08.597139 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.597121    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cmsvx"
Jul 17 21:08:08.651524 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.651494    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 17 21:08:08.653876 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.653829    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-117.ec2.internal" event={"ID":"533b658e3527087c58345fec12958890","Type":"ContainerStarted","Data":"976632af211c042a60e7d3d759f2f824871916b07c0735d892feeb34a0a1ce70"}
Jul 17 21:08:08.654795 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.654773    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal" event={"ID":"dd2d9339de65db528a31e497b009a984","Type":"ContainerStarted","Data":"54ab11e8eff9f1e58c26945b7b958da189174d7b672398d7ae2f5c6f41dca9d8"}
Jul 17 21:08:08.661870 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.661845    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/867f0a7f-0397-4df5-8832-c6ee935d00a3-cni-binary-copy\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.661958 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.661890    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e02e6e63-06c8-4b13-8cbb-4b9903aa28e6-iptables-alerter-script\") pod \"iptables-alerter-2zqxd\" (UID: \"e02e6e63-06c8-4b13-8cbb-4b9903aa28e6\") " pod="openshift-network-operator/iptables-alerter-2zqxd"
Jul 17 21:08:08.661958 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.661921    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88aca6a7-04be-448e-8bc5-d351d54fc995-registration-dir\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.661958 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.661950    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf68p\" (UniqueName: \"kubernetes.io/projected/88aca6a7-04be-448e-8bc5-d351d54fc995-kube-api-access-cf68p\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.662076 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.661976    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-system-cni-dir\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.662076 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662004    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.662076 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662033    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-etc-kubernetes\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.662244 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662086    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5c629464-bbea-40de-a8d0-8d31fa115b25-konnectivity-ca\") pod \"konnectivity-agent-7hlrp\" (UID: \"5c629464-bbea-40de-a8d0-8d31fa115b25\") " pod="kube-system/konnectivity-agent-7hlrp"
Jul 17 21:08:08.662244 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662109    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-etc-kubernetes\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.662244 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662113    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-cni-binary-copy\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.662244 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662170    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret\") pod \"global-pull-secret-syncer-6bxwh\" (UID: \"710a2804-cc1e-49f5-8842-d392bb9aff26\") " pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:08.662244 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662203    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.662427 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662245    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.662427 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662277    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba2b267b-b915-414c-9ade-8aa943cfa72e-ovnkube-config\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.662427 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662304    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-sys\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.662427 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662332    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6g7s\" (UniqueName: \"kubernetes.io/projected/98470465-ed0c-4896-8871-e8d9354e3dfa-kube-api-access-g6g7s\") pod \"node-resolver-6fvtz\" (UID: \"98470465-ed0c-4896-8871-e8d9354e3dfa\") " pod="openshift-dns/node-resolver-6fvtz"
Jul 17 21:08:08.662427 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662355    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88aca6a7-04be-448e-8bc5-d351d54fc995-etc-selinux\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.662427 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662379    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-cni-bin\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.662427 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662418    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-lib-modules\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.662648 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662435    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/867f0a7f-0397-4df5-8832-c6ee935d00a3-cni-binary-copy\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.662648 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662445    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvvt9\" (UniqueName: \"kubernetes.io/projected/89bdc455-1935-456b-bc58-288a0ed415a0-kube-api-access-vvvt9\") pod \"network-metrics-daemon-hnhk2\" (UID: \"89bdc455-1935-456b-bc58-288a0ed415a0\") " pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:08.662805 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662471    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzf85\" (UniqueName: \"kubernetes.io/projected/a9d9e563-636a-4912-8557-a9d71096efb8-kube-api-access-hzf85\") pod \"node-ca-qn5pr\" (UID: \"a9d9e563-636a-4912-8557-a9d71096efb8\") " pod="openshift-image-registry/node-ca-qn5pr"
Jul 17 21:08:08.662860 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662845    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-os-release\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.662911 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662883    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88aca6a7-04be-448e-8bc5-d351d54fc995-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.662957 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662920    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-node-log\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.662957 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662947    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba2b267b-b915-414c-9ade-8aa943cfa72e-env-overrides\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.663045 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662971    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-systemd\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.663045 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.662998    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.663045 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663027    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba2b267b-b915-414c-9ade-8aa943cfa72e-ovn-node-metrics-cert\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.663186 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663054    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-tuned\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.663186 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663133    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-host-run-k8s-cni-cncf-io\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.663186 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663145    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5c629464-bbea-40de-a8d0-8d31fa115b25-konnectivity-ca\") pod \"konnectivity-agent-7hlrp\" (UID: \"5c629464-bbea-40de-a8d0-8d31fa115b25\") " pod="kube-system/konnectivity-agent-7hlrp"
Jul 17 21:08:08.663308 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663258    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-os-release\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.663308 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663280    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-host-run-k8s-cni-cncf-io\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.663396 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663310    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-host-var-lib-cni-multus\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.663396 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663342    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4xg\" (UniqueName: \"kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg\") pod \"network-check-target-kphjs\" (UID: \"3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca\") " pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:08.663535 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663430    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-host-var-lib-cni-multus\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.663535 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663469    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-sysctl-d\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.663535 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-multus-cni-dir\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.663712 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663631    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-multus-socket-dir-parent\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.663762 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663746    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-multus-cni-dir\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.663823 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663765    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-multus-socket-dir-parent\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.663823 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663788    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-run-ovn\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.663918 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663822    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-systemd-units\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.663918 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663855    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9d9e563-636a-4912-8557-a9d71096efb8-serviceca\") pod \"node-ca-qn5pr\" (UID: \"a9d9e563-636a-4912-8557-a9d71096efb8\") " pod="openshift-image-registry/node-ca-qn5pr"
Jul 17 21:08:08.663918 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663879    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-cnibin\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.663918 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663906    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-hostroot\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.664087 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663936    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/867f0a7f-0397-4df5-8832-c6ee935d00a3-multus-daemon-config\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.664087 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.663973    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-kubelet\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.664087 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664000    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-kubernetes\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.664087 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664022    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-host\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.664087 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664051    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mslzw\" (UniqueName: \"kubernetes.io/projected/e02e6e63-06c8-4b13-8cbb-4b9903aa28e6-kube-api-access-mslzw\") pod \"iptables-alerter-2zqxd\" (UID: \"e02e6e63-06c8-4b13-8cbb-4b9903aa28e6\") " pod="openshift-network-operator/iptables-alerter-2zqxd"
Jul 17 21:08:08.664087 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664079    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-run-systemd\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.664359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664107    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h58rg\" (UniqueName: \"kubernetes.io/projected/30c718e8-f47c-4b2a-9d45-2ef525414b82-kube-api-access-h58rg\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.664359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664136    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs\") pod \"network-metrics-daemon-hnhk2\" (UID: \"89bdc455-1935-456b-bc58-288a0ed415a0\") " pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:08.664359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664161    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98470465-ed0c-4896-8871-e8d9354e3dfa-tmp-dir\") pod \"node-resolver-6fvtz\" (UID: \"98470465-ed0c-4896-8871-e8d9354e3dfa\") " pod="openshift-dns/node-resolver-6fvtz"
Jul 17 21:08:08.664359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664197    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-host-run-netns\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.664359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664231    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-host-var-lib-kubelet\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.664359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664259    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/98470465-ed0c-4896-8871-e8d9354e3dfa-hosts-file\") pod \"node-resolver-6fvtz\" (UID: \"98470465-ed0c-4896-8871-e8d9354e3dfa\") " pod="openshift-dns/node-resolver-6fvtz"
Jul 17 21:08:08.664359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664283    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88aca6a7-04be-448e-8bc5-d351d54fc995-sys-fs\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.664359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664312    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-os-release\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.664359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664338    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/710a2804-cc1e-49f5-8842-d392bb9aff26-dbus\") pod \"global-pull-secret-syncer-6bxwh\" (UID: \"710a2804-cc1e-49f5-8842-d392bb9aff26\") " pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:08.664743 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664365    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-etc-openvswitch\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.664743 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664394    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-multus-conf-dir\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.664743 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664438    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-host-run-multus-certs\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.664743 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664468    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.664743 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664495    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-slash\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.664743 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664523    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-run-netns\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.664743 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664556    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-run-openvswitch\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.664743 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664579    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-log-socket\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.664743 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664606    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-sysconfig\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.664743 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664636    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-host-var-lib-cni-bin\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.664743 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664666    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-var-lib-openvswitch\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.664743 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664690    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-modprobe-d\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.664743 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664728    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-system-cni-dir\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.665298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664754    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tkkc\" (UniqueName: \"kubernetes.io/projected/867f0a7f-0397-4df5-8832-c6ee935d00a3-kube-api-access-2tkkc\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.665298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664783    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5c629464-bbea-40de-a8d0-8d31fa115b25-agent-certs\") pod \"konnectivity-agent-7hlrp\" (UID: \"5c629464-bbea-40de-a8d0-8d31fa115b25\") " pod="kube-system/konnectivity-agent-7hlrp"
Jul 17 21:08:08.665298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664813    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88aca6a7-04be-448e-8bc5-d351d54fc995-socket-dir\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.665298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664842    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-cni-netd\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.665298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664881    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-sysctl-conf\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.665298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664904    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88aca6a7-04be-448e-8bc5-d351d54fc995-device-dir\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.665298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664936    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-cnibin\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.665298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.664990    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/710a2804-cc1e-49f5-8842-d392bb9aff26-kubelet-config\") pod \"global-pull-secret-syncer-6bxwh\" (UID: \"710a2804-cc1e-49f5-8842-d392bb9aff26\") " pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:08.665298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.665018    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba2b267b-b915-414c-9ade-8aa943cfa72e-ovnkube-script-lib\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.665298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.665044    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30c718e8-f47c-4b2a-9d45-2ef525414b82-tmp\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.665298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.665066    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e02e6e63-06c8-4b13-8cbb-4b9903aa28e6-host-slash\") pod \"iptables-alerter-2zqxd\" (UID: \"e02e6e63-06c8-4b13-8cbb-4b9903aa28e6\") " pod="openshift-network-operator/iptables-alerter-2zqxd"
Jul 17 21:08:08.665298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.665096    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnbkq\" (UniqueName: \"kubernetes.io/projected/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-kube-api-access-wnbkq\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.665298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.665125    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2jzq\" (UniqueName: \"kubernetes.io/projected/ba2b267b-b915-414c-9ade-8aa943cfa72e-kube-api-access-b2jzq\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.665298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.665150    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-run\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.665903 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.665174    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-var-lib-kubelet\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.665903 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.665205    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9d9e563-636a-4912-8557-a9d71096efb8-host\") pod \"node-ca-qn5pr\" (UID: \"a9d9e563-636a-4912-8557-a9d71096efb8\") " pod="openshift-image-registry/node-ca-qn5pr"
Jul 17 21:08:08.665903 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.665308    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9d9e563-636a-4912-8557-a9d71096efb8-host\") pod \"node-ca-qn5pr\" (UID: \"a9d9e563-636a-4912-8557-a9d71096efb8\") " pod="openshift-image-registry/node-ca-qn5pr"
Jul 17 21:08:08.665903 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.665741    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9d9e563-636a-4912-8557-a9d71096efb8-serviceca\") pod \"node-ca-qn5pr\" (UID: \"a9d9e563-636a-4912-8557-a9d71096efb8\") " pod="openshift-image-registry/node-ca-qn5pr"
Jul 17 21:08:08.665903 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.665815    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-cnibin\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.665903 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.665862    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-hostroot\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.666159 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.666099    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-system-cni-dir\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.666207 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.666153    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-host-var-lib-cni-bin\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.666207 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.666161    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/98470465-ed0c-4896-8871-e8d9354e3dfa-hosts-file\") pod \"node-resolver-6fvtz\" (UID: \"98470465-ed0c-4896-8871-e8d9354e3dfa\") " pod="openshift-dns/node-resolver-6fvtz"
Jul 17 21:08:08.666207 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.666175    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-host-run-netns\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.666341 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.666267    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-host-var-lib-kubelet\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.666391 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.666340    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-multus-conf-dir\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.666391 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.666386    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/867f0a7f-0397-4df5-8832-c6ee935d00a3-multus-daemon-config\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.666510 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.666471    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/867f0a7f-0397-4df5-8832-c6ee935d00a3-host-run-multus-certs\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.667106 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.667080    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 17 21:08:08.667463 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.667444    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98470465-ed0c-4896-8871-e8d9354e3dfa-tmp-dir\") pod \"node-resolver-6fvtz\" (UID: \"98470465-ed0c-4896-8871-e8d9354e3dfa\") " pod="openshift-dns/node-resolver-6fvtz"
Jul 17 21:08:08.670303 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.670286    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5c629464-bbea-40de-a8d0-8d31fa115b25-agent-certs\") pod \"konnectivity-agent-7hlrp\" (UID: \"5c629464-bbea-40de-a8d0-8d31fa115b25\") " pod="kube-system/konnectivity-agent-7hlrp"
Jul 17 21:08:08.671082 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:08.671051    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:08:08.671082 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:08.671070    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:08:08.671082 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:08.671081    2569 projected.go:196] Error preparing data for projected volume kube-api-access-hs4xg for pod openshift-network-diagnostics/network-check-target-kphjs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:08.671237 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:08.671128    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg podName:3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca nodeName:}" failed. No retries permitted until 2026-07-17 21:08:09.171112955 +0000 UTC m=+2.930679630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hs4xg" (UniqueName: "kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg") pod "network-check-target-kphjs" (UID: "3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:08.673190 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.673164    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tkkc\" (UniqueName: \"kubernetes.io/projected/867f0a7f-0397-4df5-8832-c6ee935d00a3-kube-api-access-2tkkc\") pod \"multus-djkwb\" (UID: \"867f0a7f-0397-4df5-8832-c6ee935d00a3\") " pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.673332 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.673313    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6g7s\" (UniqueName: \"kubernetes.io/projected/98470465-ed0c-4896-8871-e8d9354e3dfa-kube-api-access-g6g7s\") pod \"node-resolver-6fvtz\" (UID: \"98470465-ed0c-4896-8871-e8d9354e3dfa\") " pod="openshift-dns/node-resolver-6fvtz"
Jul 17 21:08:08.673559 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.673542    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzf85\" (UniqueName: \"kubernetes.io/projected/a9d9e563-636a-4912-8557-a9d71096efb8-kube-api-access-hzf85\") pod \"node-ca-qn5pr\" (UID: \"a9d9e563-636a-4912-8557-a9d71096efb8\") " pod="openshift-image-registry/node-ca-qn5pr"
Jul 17 21:08:08.766047 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766015    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf68p\" (UniqueName: \"kubernetes.io/projected/88aca6a7-04be-448e-8bc5-d351d54fc995-kube-api-access-cf68p\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.766047 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766048    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-system-cni-dir\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.766262 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766072    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.766262 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766141    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-system-cni-dir\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.766262 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766139    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-cni-binary-copy\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.766262 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766184    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret\") pod \"global-pull-secret-syncer-6bxwh\" (UID: \"710a2804-cc1e-49f5-8842-d392bb9aff26\") " pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:08.766262 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766209    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.766262 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766234    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.766262 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766258    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba2b267b-b915-414c-9ade-8aa943cfa72e-ovnkube-config\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.766606 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766282    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-sys\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.766606 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:08.766285    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:08.766606 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766339    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.766606 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:08.766362    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret podName:710a2804-cc1e-49f5-8842-d392bb9aff26 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:09.26634353 +0000 UTC m=+3.025910211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret") pod "global-pull-secret-syncer-6bxwh" (UID: "710a2804-cc1e-49f5-8842-d392bb9aff26") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:08.766606 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766389    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88aca6a7-04be-448e-8bc5-d351d54fc995-etc-selinux\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.766606 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766441    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-cni-bin\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.766606 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766464    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-lib-modules\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.766606 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766489    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvvt9\" (UniqueName: \"kubernetes.io/projected/89bdc455-1935-456b-bc58-288a0ed415a0-kube-api-access-vvvt9\") pod \"network-metrics-daemon-hnhk2\" (UID: \"89bdc455-1935-456b-bc58-288a0ed415a0\") " pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:08.766606 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766461    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.766606 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766515    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88aca6a7-04be-448e-8bc5-d351d54fc995-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.766606 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766538    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-cni-bin\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.766606 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766551    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-node-log\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.766606 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766559    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88aca6a7-04be-448e-8bc5-d351d54fc995-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.766606 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766546    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88aca6a7-04be-448e-8bc5-d351d54fc995-etc-selinux\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.767280 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766570    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba2b267b-b915-414c-9ade-8aa943cfa72e-env-overrides\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.767280 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766602    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-systemd\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.767280 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766637    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-lib-modules\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.767280 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766636    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.767280 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766685    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba2b267b-b915-414c-9ade-8aa943cfa72e-ovn-node-metrics-cert\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.767280 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766695    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-sys\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.767280 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766710    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-tuned\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.767280 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766740    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-node-log\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.767280 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766753    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-sysctl-d\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.767280 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766813    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.767280 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766836    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-systemd\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.767280 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766851    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-sysctl-d\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.767280 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766873    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba2b267b-b915-414c-9ade-8aa943cfa72e-ovnkube-config\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.767280 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766877    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-run-ovn\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.767280 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766905    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-systemd-units\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.768018 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766932    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-kubelet\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.768018 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766973    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-run-ovn\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.768018 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.766958    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-kubernetes\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.768018 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767009    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-host\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.768018 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767016    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-systemd-units\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.768018 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767043    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mslzw\" (UniqueName: \"kubernetes.io/projected/e02e6e63-06c8-4b13-8cbb-4b9903aa28e6-kube-api-access-mslzw\") pod \"iptables-alerter-2zqxd\" (UID: \"e02e6e63-06c8-4b13-8cbb-4b9903aa28e6\") " pod="openshift-network-operator/iptables-alerter-2zqxd"
Jul 17 21:08:08.768018 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767063    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-host\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.768018 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767069    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-run-systemd\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.768018 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767083    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba2b267b-b915-414c-9ade-8aa943cfa72e-env-overrides\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.768018 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767093    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h58rg\" (UniqueName: \"kubernetes.io/projected/30c718e8-f47c-4b2a-9d45-2ef525414b82-kube-api-access-h58rg\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.768018 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767099    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-kubelet\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.768018 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767121    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs\") pod \"network-metrics-daemon-hnhk2\" (UID: \"89bdc455-1935-456b-bc58-288a0ed415a0\") " pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:08.768018 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767159    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-run-systemd\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.768018 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:08.767218    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:08.768018 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767217    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.768770 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767250    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-cni-binary-copy\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.768770 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:08.767271    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs podName:89bdc455-1935-456b-bc58-288a0ed415a0 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:09.267256553 +0000 UTC m=+3.026823230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs") pod "network-metrics-daemon-hnhk2" (UID: "89bdc455-1935-456b-bc58-288a0ed415a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:08.768770 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767276    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-kubernetes\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.768770 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767312    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88aca6a7-04be-448e-8bc5-d351d54fc995-sys-fs\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.768770 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767337    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-os-release\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.768770 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767361    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/710a2804-cc1e-49f5-8842-d392bb9aff26-dbus\") pod \"global-pull-secret-syncer-6bxwh\" (UID: \"710a2804-cc1e-49f5-8842-d392bb9aff26\") " pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:08.768770 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767392    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-etc-openvswitch\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.768770 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767418    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88aca6a7-04be-448e-8bc5-d351d54fc995-sys-fs\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.768770 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767439    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.768770 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767459    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-slash\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.768770 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767464    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-etc-openvswitch\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.768770 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767441    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/710a2804-cc1e-49f5-8842-d392bb9aff26-dbus\") pod \"global-pull-secret-syncer-6bxwh\" (UID: \"710a2804-cc1e-49f5-8842-d392bb9aff26\") " pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:08.768770 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767475    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-run-netns\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.769359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767484    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-os-release\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.769359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767509    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-slash\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.769359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-run-openvswitch\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.769359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767543    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-log-socket\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.769359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767549    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-run-openvswitch\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.769359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767509    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-run-netns\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.769359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767565    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-sysconfig\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.769359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767593    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-var-lib-openvswitch\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.769359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767601    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-log-socket\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.769359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767617    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-modprobe-d\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.769359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767651    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88aca6a7-04be-448e-8bc5-d351d54fc995-socket-dir\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.769359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767679    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-sysconfig\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.769359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767683    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.769359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767694    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-cni-netd\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.769784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767688    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-var-lib-openvswitch\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.769784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767727    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-sysctl-conf\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.769784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767740    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba2b267b-b915-414c-9ade-8aa943cfa72e-host-cni-netd\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.769784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767754    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88aca6a7-04be-448e-8bc5-d351d54fc995-device-dir\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.769784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767761    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-modprobe-d\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.769784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767783    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-cnibin\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.769784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767796    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88aca6a7-04be-448e-8bc5-d351d54fc995-socket-dir\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.769784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767807    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88aca6a7-04be-448e-8bc5-d351d54fc995-device-dir\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.769784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767810    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/710a2804-cc1e-49f5-8842-d392bb9aff26-kubelet-config\") pod \"global-pull-secret-syncer-6bxwh\" (UID: \"710a2804-cc1e-49f5-8842-d392bb9aff26\") " pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:08.769784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767839    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-cnibin\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.769784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767841    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba2b267b-b915-414c-9ade-8aa943cfa72e-ovnkube-script-lib\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.769784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767857    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/710a2804-cc1e-49f5-8842-d392bb9aff26-kubelet-config\") pod \"global-pull-secret-syncer-6bxwh\" (UID: \"710a2804-cc1e-49f5-8842-d392bb9aff26\") " pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:08.769784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767867    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30c718e8-f47c-4b2a-9d45-2ef525414b82-tmp\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.769784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767894    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e02e6e63-06c8-4b13-8cbb-4b9903aa28e6-host-slash\") pod \"iptables-alerter-2zqxd\" (UID: \"e02e6e63-06c8-4b13-8cbb-4b9903aa28e6\") " pod="openshift-network-operator/iptables-alerter-2zqxd"
Jul 17 21:08:08.770278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767919    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnbkq\" (UniqueName: \"kubernetes.io/projected/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-kube-api-access-wnbkq\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.770278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767857    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-sysctl-conf\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.770278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767961    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2jzq\" (UniqueName: \"kubernetes.io/projected/ba2b267b-b915-414c-9ade-8aa943cfa72e-kube-api-access-b2jzq\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.770278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767965    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e02e6e63-06c8-4b13-8cbb-4b9903aa28e6-host-slash\") pod \"iptables-alerter-2zqxd\" (UID: \"e02e6e63-06c8-4b13-8cbb-4b9903aa28e6\") " pod="openshift-network-operator/iptables-alerter-2zqxd"
Jul 17 21:08:08.770278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.767986    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-run\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.770278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.768021    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-var-lib-kubelet\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.770278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.768051    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e02e6e63-06c8-4b13-8cbb-4b9903aa28e6-iptables-alerter-script\") pod \"iptables-alerter-2zqxd\" (UID: \"e02e6e63-06c8-4b13-8cbb-4b9903aa28e6\") " pod="openshift-network-operator/iptables-alerter-2zqxd"
Jul 17 21:08:08.770278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.768075    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88aca6a7-04be-448e-8bc5-d351d54fc995-registration-dir\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.770278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.768160    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88aca6a7-04be-448e-8bc5-d351d54fc995-registration-dir\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.770278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.768165    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-run\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.770278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.768185    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30c718e8-f47c-4b2a-9d45-2ef525414b82-var-lib-kubelet\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.770278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.768315    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba2b267b-b915-414c-9ade-8aa943cfa72e-ovnkube-script-lib\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.770278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.768674    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e02e6e63-06c8-4b13-8cbb-4b9903aa28e6-iptables-alerter-script\") pod \"iptables-alerter-2zqxd\" (UID: \"e02e6e63-06c8-4b13-8cbb-4b9903aa28e6\") " pod="openshift-network-operator/iptables-alerter-2zqxd"
Jul 17 21:08:08.770278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.769440    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba2b267b-b915-414c-9ade-8aa943cfa72e-ovn-node-metrics-cert\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.770713 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.769617    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/30c718e8-f47c-4b2a-9d45-2ef525414b82-etc-tuned\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.770713 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.769982    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30c718e8-f47c-4b2a-9d45-2ef525414b82-tmp\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.776960 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.776907    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf68p\" (UniqueName: \"kubernetes.io/projected/88aca6a7-04be-448e-8bc5-d351d54fc995-kube-api-access-cf68p\") pod \"aws-ebs-csi-driver-node-kxbgg\" (UID: \"88aca6a7-04be-448e-8bc5-d351d54fc995\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.776960 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.776916    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvvt9\" (UniqueName: \"kubernetes.io/projected/89bdc455-1935-456b-bc58-288a0ed415a0-kube-api-access-vvvt9\") pod \"network-metrics-daemon-hnhk2\" (UID: \"89bdc455-1935-456b-bc58-288a0ed415a0\") " pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:08.777647 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.777625    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2jzq\" (UniqueName: \"kubernetes.io/projected/ba2b267b-b915-414c-9ade-8aa943cfa72e-kube-api-access-b2jzq\") pod \"ovnkube-node-4ccjx\" (UID: \"ba2b267b-b915-414c-9ade-8aa943cfa72e\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.777836 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.777819    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mslzw\" (UniqueName: \"kubernetes.io/projected/e02e6e63-06c8-4b13-8cbb-4b9903aa28e6-kube-api-access-mslzw\") pod \"iptables-alerter-2zqxd\" (UID: \"e02e6e63-06c8-4b13-8cbb-4b9903aa28e6\") " pod="openshift-network-operator/iptables-alerter-2zqxd"
Jul 17 21:08:08.778775 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.778758    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h58rg\" (UniqueName: \"kubernetes.io/projected/30c718e8-f47c-4b2a-9d45-2ef525414b82-kube-api-access-h58rg\") pod \"tuned-5lckv\" (UID: \"30c718e8-f47c-4b2a-9d45-2ef525414b82\") " pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.778883 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.778791    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnbkq\" (UniqueName: \"kubernetes.io/projected/9dc91e48-32c6-42ad-b3e9-7be1ec357d6f-kube-api-access-wnbkq\") pod \"multus-additional-cni-plugins-hxjhv\" (UID: \"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f\") " pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.846183 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.846101    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:08:08.853065 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.853034    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6fvtz"
Jul 17 21:08:08.861445 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.861426    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qn5pr"
Jul 17 21:08:08.870248 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.870228    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-djkwb"
Jul 17 21:08:08.875873 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.875857    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7hlrp"
Jul 17 21:08:08.882446 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.882428    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2zqxd"
Jul 17 21:08:08.887057 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.887042    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg"
Jul 17 21:08:08.892563 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.892540    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hxjhv"
Jul 17 21:08:08.898677 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.898659    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5lckv"
Jul 17 21:08:08.902233 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:08.902217    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:08.980066 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:08.980035    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98470465_ed0c_4896_8871_e8d9354e3dfa.slice/crio-390e5e9149838f15fd7a0d6e2e089d1b9d3107b454116ef67623ac2f4da42673 WatchSource:0}: Error finding container 390e5e9149838f15fd7a0d6e2e089d1b9d3107b454116ef67623ac2f4da42673: Status 404 returned error can't find the container with id 390e5e9149838f15fd7a0d6e2e089d1b9d3107b454116ef67623ac2f4da42673
Jul 17 21:08:08.980872 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:08.980847    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc91e48_32c6_42ad_b3e9_7be1ec357d6f.slice/crio-191ec145ad544b72631b614708db83c84bca72154a9ecc4415b30b25f5438a47 WatchSource:0}: Error finding container 191ec145ad544b72631b614708db83c84bca72154a9ecc4415b30b25f5438a47: Status 404 returned error can't find the container with id 191ec145ad544b72631b614708db83c84bca72154a9ecc4415b30b25f5438a47
Jul 17 21:08:08.983900 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:08.983880    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2b267b_b915_414c_9ade_8aa943cfa72e.slice/crio-406bc4f1c600cde683270e41bede9d73966e1ceea7a36a989b79cdcc9e43e021 WatchSource:0}: Error finding container 406bc4f1c600cde683270e41bede9d73966e1ceea7a36a989b79cdcc9e43e021: Status 404 returned error can't find the container with id 406bc4f1c600cde683270e41bede9d73966e1ceea7a36a989b79cdcc9e43e021
Jul 17 21:08:08.986846 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:08.986817    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88aca6a7_04be_448e_8bc5_d351d54fc995.slice/crio-561143f77df27cef2dcb561bcf02626a9369c52a53288abfdfe1cc5dfa5991ce WatchSource:0}: Error finding container 561143f77df27cef2dcb561bcf02626a9369c52a53288abfdfe1cc5dfa5991ce: Status 404 returned error can't find the container with id 561143f77df27cef2dcb561bcf02626a9369c52a53288abfdfe1cc5dfa5991ce
Jul 17 21:08:08.987599 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:08.987514    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c629464_bbea_40de_a8d0_8d31fa115b25.slice/crio-1e1b60d6a13c9073f94939e66942a11c93f9f25880af1693b88c2067789074c0 WatchSource:0}: Error finding container 1e1b60d6a13c9073f94939e66942a11c93f9f25880af1693b88c2067789074c0: Status 404 returned error can't find the container with id 1e1b60d6a13c9073f94939e66942a11c93f9f25880af1693b88c2067789074c0
Jul 17 21:08:08.988643 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:08.988619    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867f0a7f_0397_4df5_8832_c6ee935d00a3.slice/crio-8fdb02756e46d864159489f1d76db49c4f9159f8d9a0256afe31f55be54e7a04 WatchSource:0}: Error finding container 8fdb02756e46d864159489f1d76db49c4f9159f8d9a0256afe31f55be54e7a04: Status 404 returned error can't find the container with id 8fdb02756e46d864159489f1d76db49c4f9159f8d9a0256afe31f55be54e7a04
Jul 17 21:08:08.989434 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:08.989382    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode02e6e63_06c8_4b13_8cbb_4b9903aa28e6.slice/crio-c999d38744880ac878c8bd823e2218d888cf9ae498076c81812011295ae150c3 WatchSource:0}: Error finding container c999d38744880ac878c8bd823e2218d888cf9ae498076c81812011295ae150c3: Status 404 returned error can't find the container with id c999d38744880ac878c8bd823e2218d888cf9ae498076c81812011295ae150c3
Jul 17 21:08:09.010478 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:09.010456    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d9e563_636a_4912_8557_a9d71096efb8.slice/crio-5a466ce1d5ae2f4cbaa0c7effbb95704c3f4aa8a1820c28e538f8810147946a1 WatchSource:0}: Error finding container 5a466ce1d5ae2f4cbaa0c7effbb95704c3f4aa8a1820c28e538f8810147946a1: Status 404 returned error can't find the container with id 5a466ce1d5ae2f4cbaa0c7effbb95704c3f4aa8a1820c28e538f8810147946a1
Jul 17 21:08:09.011331 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:09.011310    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c718e8_f47c_4b2a_9d45_2ef525414b82.slice/crio-ceae4e9484c714c9d899301a1d8493eb67c033ea5dd43fe7543ddc480e9ebfcb WatchSource:0}: Error finding container ceae4e9484c714c9d899301a1d8493eb67c033ea5dd43fe7543ddc480e9ebfcb: Status 404 returned error can't find the container with id ceae4e9484c714c9d899301a1d8493eb67c033ea5dd43fe7543ddc480e9ebfcb
Jul 17 21:08:09.172312 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:09.172197    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4xg\" (UniqueName: \"kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg\") pod \"network-check-target-kphjs\" (UID: \"3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca\") " pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:09.172519 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:09.172330    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:08:09.172519 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:09.172351    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:08:09.172519 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:09.172363    2569 projected.go:196] Error preparing data for projected volume kube-api-access-hs4xg for pod openshift-network-diagnostics/network-check-target-kphjs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:09.172519 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:09.172444    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg podName:3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca nodeName:}" failed. No retries permitted until 2026-07-17 21:08:10.172422358 +0000 UTC m=+3.931989031 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hs4xg" (UniqueName: "kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg") pod "network-check-target-kphjs" (UID: "3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:09.273653 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:09.273569    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret\") pod \"global-pull-secret-syncer-6bxwh\" (UID: \"710a2804-cc1e-49f5-8842-d392bb9aff26\") " pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:09.273808 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:09.273751    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs\") pod \"network-metrics-daemon-hnhk2\" (UID: \"89bdc455-1935-456b-bc58-288a0ed415a0\") " pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:09.273808 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:09.273781    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:09.273910 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:09.273852    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret podName:710a2804-cc1e-49f5-8842-d392bb9aff26 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:10.273830892 +0000 UTC m=+4.033397569 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret") pod "global-pull-secret-syncer-6bxwh" (UID: "710a2804-cc1e-49f5-8842-d392bb9aff26") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:09.273910 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:09.273884    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:09.274011 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:09.273944    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs podName:89bdc455-1935-456b-bc58-288a0ed415a0 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:10.273927015 +0000 UTC m=+4.033493698 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs") pod "network-metrics-daemon-hnhk2" (UID: "89bdc455-1935-456b-bc58-288a0ed415a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:09.600771 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:09.600645    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 21:03:08 +0000 UTC" deadline="2028-03-20 16:17:12.705658926 +0000 UTC"
Jul 17 21:08:09.600771 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:09.600716    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14683h9m3.104947594s"
Jul 17 21:08:09.660786 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:09.660608    2569 generic.go:356] "Generic (PLEG): container finished" podID="dd2d9339de65db528a31e497b009a984" containerID="3bf74fab8f3bbdf19c3ed11b54cced5f78445f916434f9a39e9d2c6b2dbe0004" exitCode=0
Jul 17 21:08:09.660786 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:09.660697    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal" event={"ID":"dd2d9339de65db528a31e497b009a984","Type":"ContainerDied","Data":"3bf74fab8f3bbdf19c3ed11b54cced5f78445f916434f9a39e9d2c6b2dbe0004"}
Jul 17 21:08:09.665368 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:09.665285    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5lckv" event={"ID":"30c718e8-f47c-4b2a-9d45-2ef525414b82","Type":"ContainerStarted","Data":"ceae4e9484c714c9d899301a1d8493eb67c033ea5dd43fe7543ddc480e9ebfcb"}
Jul 17 21:08:09.667288 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:09.667211    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2zqxd" event={"ID":"e02e6e63-06c8-4b13-8cbb-4b9903aa28e6","Type":"ContainerStarted","Data":"c999d38744880ac878c8bd823e2218d888cf9ae498076c81812011295ae150c3"}
Jul 17 21:08:09.669127 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:09.669086    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-djkwb" event={"ID":"867f0a7f-0397-4df5-8832-c6ee935d00a3","Type":"ContainerStarted","Data":"8fdb02756e46d864159489f1d76db49c4f9159f8d9a0256afe31f55be54e7a04"}
Jul 17 21:08:09.670450 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:09.670380    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7hlrp" event={"ID":"5c629464-bbea-40de-a8d0-8d31fa115b25","Type":"ContainerStarted","Data":"1e1b60d6a13c9073f94939e66942a11c93f9f25880af1693b88c2067789074c0"}
Jul 17 21:08:09.672725 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:09.672658    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg" event={"ID":"88aca6a7-04be-448e-8bc5-d351d54fc995","Type":"ContainerStarted","Data":"561143f77df27cef2dcb561bcf02626a9369c52a53288abfdfe1cc5dfa5991ce"}
Jul 17 21:08:09.674916 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:09.674844    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxjhv" event={"ID":"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f","Type":"ContainerStarted","Data":"191ec145ad544b72631b614708db83c84bca72154a9ecc4415b30b25f5438a47"}
Jul 17 21:08:09.677354 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:09.676581    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qn5pr" event={"ID":"a9d9e563-636a-4912-8557-a9d71096efb8","Type":"ContainerStarted","Data":"5a466ce1d5ae2f4cbaa0c7effbb95704c3f4aa8a1820c28e538f8810147946a1"}
Jul 17 21:08:09.678000 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:09.677979    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx" event={"ID":"ba2b267b-b915-414c-9ade-8aa943cfa72e","Type":"ContainerStarted","Data":"406bc4f1c600cde683270e41bede9d73966e1ceea7a36a989b79cdcc9e43e021"}
Jul 17 21:08:09.680034 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:09.679983    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6fvtz" event={"ID":"98470465-ed0c-4896-8871-e8d9354e3dfa","Type":"ContainerStarted","Data":"390e5e9149838f15fd7a0d6e2e089d1b9d3107b454116ef67623ac2f4da42673"}
Jul 17 21:08:10.182439 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:10.181837    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4xg\" (UniqueName: \"kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg\") pod \"network-check-target-kphjs\" (UID: \"3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca\") " pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:10.182439 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:10.182003    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:08:10.182439 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:10.182022    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:08:10.182439 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:10.182034    2569 projected.go:196] Error preparing data for projected volume kube-api-access-hs4xg for pod openshift-network-diagnostics/network-check-target-kphjs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:10.182439 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:10.182095    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg podName:3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca nodeName:}" failed. No retries permitted until 2026-07-17 21:08:12.182079023 +0000 UTC m=+5.941645699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hs4xg" (UniqueName: "kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg") pod "network-check-target-kphjs" (UID: "3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:10.283032 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:10.282874    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs\") pod \"network-metrics-daemon-hnhk2\" (UID: \"89bdc455-1935-456b-bc58-288a0ed415a0\") " pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:10.283032 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:10.282962    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret\") pod \"global-pull-secret-syncer-6bxwh\" (UID: \"710a2804-cc1e-49f5-8842-d392bb9aff26\") " pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:10.283032 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:10.283021    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:10.283310 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:10.283080    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:10.283310 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:10.283095    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs podName:89bdc455-1935-456b-bc58-288a0ed415a0 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:12.283075104 +0000 UTC m=+6.042641790 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs") pod "network-metrics-daemon-hnhk2" (UID: "89bdc455-1935-456b-bc58-288a0ed415a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:10.283310 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:10.283139    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret podName:710a2804-cc1e-49f5-8842-d392bb9aff26 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:12.283123016 +0000 UTC m=+6.042689699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret") pod "global-pull-secret-syncer-6bxwh" (UID: "710a2804-cc1e-49f5-8842-d392bb9aff26") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:10.601822 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:10.601733    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 21:03:08 +0000 UTC" deadline="2028-02-23 17:54:14.490926556 +0000 UTC"
Jul 17 21:08:10.601822 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:10.601774    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14060h46m3.889156802s"
Jul 17 21:08:10.658526 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:10.649884    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:10.658526 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:10.650009    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bxwh" podUID="710a2804-cc1e-49f5-8842-d392bb9aff26"
Jul 17 21:08:10.658526 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:10.650374    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:10.658526 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:10.650485    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kphjs" podUID="3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca"
Jul 17 21:08:10.658526 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:10.650551    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:10.658526 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:10.650626    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhk2" podUID="89bdc455-1935-456b-bc58-288a0ed415a0"
Jul 17 21:08:12.199980 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:12.199313    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4xg\" (UniqueName: \"kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg\") pod \"network-check-target-kphjs\" (UID: \"3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca\") " pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:12.199980 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:12.199534    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:08:12.199980 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:12.199556    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:08:12.199980 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:12.199568    2569 projected.go:196] Error preparing data for projected volume kube-api-access-hs4xg for pod openshift-network-diagnostics/network-check-target-kphjs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:12.199980 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:12.199620    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg podName:3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca nodeName:}" failed. No retries permitted until 2026-07-17 21:08:16.199602786 +0000 UTC m=+9.959169461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hs4xg" (UniqueName: "kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg") pod "network-check-target-kphjs" (UID: "3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:12.300727 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:12.300667    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret\") pod \"global-pull-secret-syncer-6bxwh\" (UID: \"710a2804-cc1e-49f5-8842-d392bb9aff26\") " pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:12.300906 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:12.300816    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs\") pod \"network-metrics-daemon-hnhk2\" (UID: \"89bdc455-1935-456b-bc58-288a0ed415a0\") " pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:12.300991 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:12.300948    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:12.301048 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:12.301006    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs podName:89bdc455-1935-456b-bc58-288a0ed415a0 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:16.300987562 +0000 UTC m=+10.060554233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs") pod "network-metrics-daemon-hnhk2" (UID: "89bdc455-1935-456b-bc58-288a0ed415a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:12.301433 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:12.301368    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:12.301626 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:12.301608    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret podName:710a2804-cc1e-49f5-8842-d392bb9aff26 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:16.301588719 +0000 UTC m=+10.061155390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret") pod "global-pull-secret-syncer-6bxwh" (UID: "710a2804-cc1e-49f5-8842-d392bb9aff26") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:12.660002 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:12.657011    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:12.660002 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:12.657161    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bxwh" podUID="710a2804-cc1e-49f5-8842-d392bb9aff26"
Jul 17 21:08:12.660002 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:12.657613    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:12.660002 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:12.657729    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kphjs" podUID="3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca"
Jul 17 21:08:12.660002 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:12.657796    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:12.660002 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:12.657904    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhk2" podUID="89bdc455-1935-456b-bc58-288a0ed415a0"
Jul 17 21:08:14.650803 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:14.650775    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:14.651213 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:14.650804    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:14.651213 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:14.650882    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:14.651213 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:14.650901    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kphjs" podUID="3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca"
Jul 17 21:08:14.651213 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:14.650994    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhk2" podUID="89bdc455-1935-456b-bc58-288a0ed415a0"
Jul 17 21:08:14.651213 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:14.651073    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bxwh" podUID="710a2804-cc1e-49f5-8842-d392bb9aff26"
Jul 17 21:08:16.235130 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:16.235087    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4xg\" (UniqueName: \"kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg\") pod \"network-check-target-kphjs\" (UID: \"3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca\") " pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:16.235566 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:16.235301    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:08:16.235566 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:16.235342    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:08:16.235566 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:16.235356    2569 projected.go:196] Error preparing data for projected volume kube-api-access-hs4xg for pod openshift-network-diagnostics/network-check-target-kphjs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:16.235566 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:16.235432    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg podName:3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca nodeName:}" failed. No retries permitted until 2026-07-17 21:08:24.235396678 +0000 UTC m=+17.994963388 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hs4xg" (UniqueName: "kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg") pod "network-check-target-kphjs" (UID: "3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:16.336041 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:16.336001    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs\") pod \"network-metrics-daemon-hnhk2\" (UID: \"89bdc455-1935-456b-bc58-288a0ed415a0\") " pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:16.336214 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:16.336120    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret\") pod \"global-pull-secret-syncer-6bxwh\" (UID: \"710a2804-cc1e-49f5-8842-d392bb9aff26\") " pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:16.336214 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:16.336128    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:16.336214 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:16.336196    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs podName:89bdc455-1935-456b-bc58-288a0ed415a0 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:24.336175939 +0000 UTC m=+18.095742625 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs") pod "network-metrics-daemon-hnhk2" (UID: "89bdc455-1935-456b-bc58-288a0ed415a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:16.336386 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:16.336238    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:16.336386 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:16.336304    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret podName:710a2804-cc1e-49f5-8842-d392bb9aff26 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:24.336285534 +0000 UTC m=+18.095852219 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret") pod "global-pull-secret-syncer-6bxwh" (UID: "710a2804-cc1e-49f5-8842-d392bb9aff26") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:16.651241 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:16.651167    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:16.651453 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:16.651268    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bxwh" podUID="710a2804-cc1e-49f5-8842-d392bb9aff26"
Jul 17 21:08:16.651643 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:16.651611    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:16.651749 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:16.651732    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kphjs" podUID="3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca"
Jul 17 21:08:16.651803 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:16.651783    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:16.651907 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:16.651878    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhk2" podUID="89bdc455-1935-456b-bc58-288a0ed415a0"
Jul 17 21:08:18.654370 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:18.654338    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:18.654833 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:18.654378    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:18.654833 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:18.654347    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:18.654833 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:18.654487    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bxwh" podUID="710a2804-cc1e-49f5-8842-d392bb9aff26"
Jul 17 21:08:18.654833 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:18.654625    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kphjs" podUID="3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca"
Jul 17 21:08:18.654833 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:18.654722    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhk2" podUID="89bdc455-1935-456b-bc58-288a0ed415a0"
Jul 17 21:08:20.650328 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:20.650294    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:20.650328 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:20.650311    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:20.650817 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:20.650423    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kphjs" podUID="3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca"
Jul 17 21:08:20.650817 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:20.650493    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhk2" podUID="89bdc455-1935-456b-bc58-288a0ed415a0"
Jul 17 21:08:20.650817 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:20.650531    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:20.650817 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:20.650632    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bxwh" podUID="710a2804-cc1e-49f5-8842-d392bb9aff26"
Jul 17 21:08:22.650363 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:22.650327    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:22.650844 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:22.650327    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:22.650844 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:22.650470    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bxwh" podUID="710a2804-cc1e-49f5-8842-d392bb9aff26"
Jul 17 21:08:22.650844 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:22.650569    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhk2" podUID="89bdc455-1935-456b-bc58-288a0ed415a0"
Jul 17 21:08:22.650844 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:22.650327    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:22.650844 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:22.650635    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kphjs" podUID="3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca"
Jul 17 21:08:24.293690 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:24.293654    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4xg\" (UniqueName: \"kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg\") pod \"network-check-target-kphjs\" (UID: \"3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca\") " pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:24.294117 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:24.293784    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:08:24.294117 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:24.293805    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:08:24.294117 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:24.293815    2569 projected.go:196] Error preparing data for projected volume kube-api-access-hs4xg for pod openshift-network-diagnostics/network-check-target-kphjs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:24.294117 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:24.293884    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg podName:3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca nodeName:}" failed. No retries permitted until 2026-07-17 21:08:40.293867525 +0000 UTC m=+34.053434199 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hs4xg" (UniqueName: "kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg") pod "network-check-target-kphjs" (UID: "3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:24.394633 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:24.394598    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs\") pod \"network-metrics-daemon-hnhk2\" (UID: \"89bdc455-1935-456b-bc58-288a0ed415a0\") " pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:24.394808 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:24.394654    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret\") pod \"global-pull-secret-syncer-6bxwh\" (UID: \"710a2804-cc1e-49f5-8842-d392bb9aff26\") " pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:24.394808 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:24.394761    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:24.394910 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:24.394809    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret podName:710a2804-cc1e-49f5-8842-d392bb9aff26 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:40.39479186 +0000 UTC m=+34.154358549 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret") pod "global-pull-secret-syncer-6bxwh" (UID: "710a2804-cc1e-49f5-8842-d392bb9aff26") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:24.394910 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:24.394868    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:24.395009 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:24.394956    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs podName:89bdc455-1935-456b-bc58-288a0ed415a0 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:40.39494169 +0000 UTC m=+34.154508383 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs") pod "network-metrics-daemon-hnhk2" (UID: "89bdc455-1935-456b-bc58-288a0ed415a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:24.650413 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:24.650363    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:24.650413 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:24.650384    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:24.650652 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:24.650430    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:24.650652 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:24.650532    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kphjs" podUID="3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca"
Jul 17 21:08:24.650755 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:24.650681    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhk2" podUID="89bdc455-1935-456b-bc58-288a0ed415a0"
Jul 17 21:08:24.650805 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:24.650779    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bxwh" podUID="710a2804-cc1e-49f5-8842-d392bb9aff26"
Jul 17 21:08:26.650892 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:26.650850    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:26.651321 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:26.650950    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:26.651321 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:26.650970    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kphjs" podUID="3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca"
Jul 17 21:08:26.651321 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:26.651069    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhk2" podUID="89bdc455-1935-456b-bc58-288a0ed415a0"
Jul 17 21:08:26.651321 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:26.651128    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:26.651321 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:26.651199    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bxwh" podUID="710a2804-cc1e-49f5-8842-d392bb9aff26"
Jul 17 21:08:27.731582 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:27.730892    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-117.ec2.internal" event={"ID":"533b658e3527087c58345fec12958890","Type":"ContainerStarted","Data":"1aca6897451d031578e90eadbacbad0c1e384567988c5764b9865fbbfb6ce1d2"}
Jul 17 21:08:27.745485 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:27.743615    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal" event={"ID":"dd2d9339de65db528a31e497b009a984","Type":"ContainerStarted","Data":"125c340b6dd172da634251127dfe2db91701fc7e65945d7815c9e3ee13b3198b"}
Jul 17 21:08:27.764633 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:27.756418    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7hlrp" event={"ID":"5c629464-bbea-40de-a8d0-8d31fa115b25","Type":"ContainerStarted","Data":"555c57e409ccefd7851e341464a74a65fb43c8418e8aa79b2d74d8d927730d94"}
Jul 17 21:08:27.773560 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:27.772620    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-117.ec2.internal" podStartSLOduration=20.772600126 podStartE2EDuration="20.772600126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:27.772123063 +0000 UTC m=+21.531689769" watchObservedRunningTime="2026-07-17 21:08:27.772600126 +0000 UTC m=+21.532166821"
Jul 17 21:08:27.773560 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:27.772911    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-117.ec2.internal" podStartSLOduration=20.772902721 podStartE2EDuration="20.772902721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:27.748234517 +0000 UTC m=+21.507801229" watchObservedRunningTime="2026-07-17 21:08:27.772902721 +0000 UTC m=+21.532469420"
Jul 17 21:08:27.789372 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:27.789192    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7hlrp" podStartSLOduration=12.748383562 podStartE2EDuration="21.789171348s" totalImagesPullingTime="9.040787786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:27.788131296 +0000 UTC m=+21.547697994" watchObservedRunningTime="2026-07-17 21:08:27.789171348 +0000 UTC m=+21.548738043"
Jul 17 21:08:28.650588 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:28.650376    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:28.650740 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:28.650376    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:28.650740 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:28.650686    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhk2" podUID="89bdc455-1935-456b-bc58-288a0ed415a0"
Jul 17 21:08:28.650740 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:28.650452    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:28.650887 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:28.650760    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kphjs" podUID="3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca"
Jul 17 21:08:28.650887 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:28.650816    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bxwh" podUID="710a2804-cc1e-49f5-8842-d392bb9aff26"
Jul 17 21:08:28.767849 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:28.767817    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5lckv" event={"ID":"30c718e8-f47c-4b2a-9d45-2ef525414b82","Type":"ContainerStarted","Data":"2f4e904c870b4d4a9c6c4f00b05ea80d6cab689fe58bb8a43eb7b3378cfcb163"}
Jul 17 21:08:28.769337 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:28.769310    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-djkwb" event={"ID":"867f0a7f-0397-4df5-8832-c6ee935d00a3","Type":"ContainerStarted","Data":"dac9f214e46c4454fc980f7e44fb21a90f9dc1f64392a5a0637da138bdaa2441"}
Jul 17 21:08:28.772722 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:28.772697    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg" event={"ID":"88aca6a7-04be-448e-8bc5-d351d54fc995","Type":"ContainerStarted","Data":"18d285bb305044c136245954eb770e05d0808152ad2116c7e766ff9c6144ff59"}
Jul 17 21:08:28.774149 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:28.774128    2569 generic.go:356] "Generic (PLEG): container finished" podID="9dc91e48-32c6-42ad-b3e9-7be1ec357d6f" containerID="22b0faed175e1be2ba05240f781d0d843597974c44caeeb41c273a795dd11ec2" exitCode=0
Jul 17 21:08:28.774264 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:28.774196    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxjhv" event={"ID":"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f","Type":"ContainerDied","Data":"22b0faed175e1be2ba05240f781d0d843597974c44caeeb41c273a795dd11ec2"}
Jul 17 21:08:28.775644 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:28.775617    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qn5pr" event={"ID":"a9d9e563-636a-4912-8557-a9d71096efb8","Type":"ContainerStarted","Data":"870260e46146648530b07782b7eaf2859d0a7113ee97a32c77aada25dda173e2"}
Jul 17 21:08:28.778499 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:28.778479    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx" event={"ID":"ba2b267b-b915-414c-9ade-8aa943cfa72e","Type":"ContainerStarted","Data":"a4879f7baff7b23265d973df5c10460f7e8ef33f4c45a2b6e658485fca955366"}
Jul 17 21:08:28.778597 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:28.778542    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx" event={"ID":"ba2b267b-b915-414c-9ade-8aa943cfa72e","Type":"ContainerStarted","Data":"34d56e41d6d35a36388293a1610904b2299eeb88c0f7c06d0046f40dc5c342d7"}
Jul 17 21:08:28.778597 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:28.778578    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx" event={"ID":"ba2b267b-b915-414c-9ade-8aa943cfa72e","Type":"ContainerStarted","Data":"7bc83e12d401fe5ef181a713340a88bf7dfab28f684bbdfe2cac073ec249cec7"}
Jul 17 21:08:28.778700 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:28.778621    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx" event={"ID":"ba2b267b-b915-414c-9ade-8aa943cfa72e","Type":"ContainerStarted","Data":"b94347b74effea2e255fe733d794906f447fcf66869b9aa0a0617bb7b5a00db1"}
Jul 17 21:08:28.778700 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:28.778651    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx" event={"ID":"ba2b267b-b915-414c-9ade-8aa943cfa72e","Type":"ContainerStarted","Data":"dbe7591bdd3e4deb7bd2d9bb84431e0f8459e155e0a7702aebc2505ef67f4142"}
Jul 17 21:08:28.778700 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:28.778695    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx" event={"ID":"ba2b267b-b915-414c-9ade-8aa943cfa72e","Type":"ContainerStarted","Data":"169fec5214c41c9f8808c1c3f3266e206bae6ce78a967f96d42078d19ab26b25"}
Jul 17 21:08:28.779841 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:28.779817    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6fvtz" event={"ID":"98470465-ed0c-4896-8871-e8d9354e3dfa","Type":"ContainerStarted","Data":"6afe6006faff2c2190f8c10050abd86a6f092307e6239abc70328469c7146464"}
Jul 17 21:08:28.784295 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:28.784257    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5lckv" podStartSLOduration=3.224473231 podStartE2EDuration="21.784244498s" totalImagesPullingTime="18.559771267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:28.783937091 +0000 UTC m=+22.543503786" watchObservedRunningTime="2026-07-17 21:08:28.784244498 +0000 UTC m=+22.543811192"
Jul 17 21:08:28.798859 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:28.797901    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6fvtz" podStartSLOduration=4.336212338 podStartE2EDuration="22.79788036s" totalImagesPullingTime="18.461668022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:28.796157026 +0000 UTC m=+22.555723711" watchObservedRunningTime="2026-07-17 21:08:28.79788036 +0000 UTC m=+22.557447055"
Jul 17 21:08:28.848440 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:28.848335    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qn5pr" podStartSLOduration=4.288870067 podStartE2EDuration="22.848319682s" totalImagesPullingTime="18.559449615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:28.812678889 +0000 UTC m=+22.572245604" watchObservedRunningTime="2026-07-17 21:08:28.848319682 +0000 UTC m=+22.607886399"
Jul 17 21:08:28.864363 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:28.864317    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-djkwb" podStartSLOduration=4.114504167 podStartE2EDuration="22.864303209s" totalImagesPullingTime="18.749799042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:28.864154845 +0000 UTC m=+22.623721540" watchObservedRunningTime="2026-07-17 21:08:28.864303209 +0000 UTC m=+22.623869903"
Jul 17 21:08:29.101804 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:29.101623    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 17 21:08:29.642209 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:29.642080    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T21:08:29.101638005Z","UUID":"1979906b-eccc-4e8a-afea-2856481d1603","Handler":null,"Name":"","Endpoint":""}
Jul 17 21:08:29.644132 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:29.644108    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 17 21:08:29.644268 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:29.644151    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 17 21:08:29.784836 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:29.784802    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg" event={"ID":"88aca6a7-04be-448e-8bc5-d351d54fc995","Type":"ContainerStarted","Data":"d42f899483080afb14baec2f3e3030c822297771117a4a781d1526bcfe4f7400"}
Jul 17 21:08:29.786871 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:29.786840    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2zqxd" event={"ID":"e02e6e63-06c8-4b13-8cbb-4b9903aa28e6","Type":"ContainerStarted","Data":"ac9462eb5835e52205d6a4001b8be94e37c4366b375b938293380e75b1b8d8a5"}
Jul 17 21:08:29.800795 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:29.800741    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2zqxd" podStartSLOduration=5.211265438 podStartE2EDuration="23.800725516s" totalImagesPullingTime="18.589460078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:29.800559191 +0000 UTC m=+23.560125884" watchObservedRunningTime="2026-07-17 21:08:29.800725516 +0000 UTC m=+23.560292209"
Jul 17 21:08:30.650535 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:30.650306    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:30.650800 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:30.650306    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:30.650800 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:30.650633    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bxwh" podUID="710a2804-cc1e-49f5-8842-d392bb9aff26"
Jul 17 21:08:30.650800 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:30.650307    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:30.650800 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:30.650734    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kphjs" podUID="3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca"
Jul 17 21:08:30.651298 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:30.651257    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhk2" podUID="89bdc455-1935-456b-bc58-288a0ed415a0"
Jul 17 21:08:30.792426 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:30.792367    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg" event={"ID":"88aca6a7-04be-448e-8bc5-d351d54fc995","Type":"ContainerStarted","Data":"a056597e3446b5c33ea826dc0bfdca27fbbae6ea3ce865eb3d675d9f4a08e750"}
Jul 17 21:08:30.796429 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:30.796389    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx" event={"ID":"ba2b267b-b915-414c-9ade-8aa943cfa72e","Type":"ContainerStarted","Data":"084fb6e60ff74f95654eaf1d956e03c5fc0f0a268e924f63fadd6dee92bc4ed7"}
Jul 17 21:08:30.810124 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:30.810077    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxbgg" podStartSLOduration=3.080863802 podStartE2EDuration="23.810058701s" totalImagesPullingTime="20.729194899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:30.809007657 +0000 UTC m=+24.568574351" watchObservedRunningTime="2026-07-17 21:08:30.810058701 +0000 UTC m=+24.569625399"
Jul 17 21:08:32.224349 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:32.224320    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7hlrp"
Jul 17 21:08:32.225602 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:32.225581    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7hlrp"
Jul 17 21:08:32.650187 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:32.650107    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:32.650187 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:32.650133    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:32.650187 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:32.650149    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:32.650508 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:32.650239    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kphjs" podUID="3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca"
Jul 17 21:08:32.650508 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:32.650339    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhk2" podUID="89bdc455-1935-456b-bc58-288a0ed415a0"
Jul 17 21:08:32.650508 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:32.650420    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bxwh" podUID="710a2804-cc1e-49f5-8842-d392bb9aff26"
Jul 17 21:08:32.800273 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:32.800195    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7hlrp"
Jul 17 21:08:32.800914 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:32.800887    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7hlrp"
Jul 17 21:08:34.650112 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:34.649934    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:34.650820 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:34.649939    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:34.650820 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:34.650203    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhk2" podUID="89bdc455-1935-456b-bc58-288a0ed415a0"
Jul 17 21:08:34.650820 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:34.649942    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:34.650820 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:34.650274    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kphjs" podUID="3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca"
Jul 17 21:08:34.650820 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:34.650354    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bxwh" podUID="710a2804-cc1e-49f5-8842-d392bb9aff26"
Jul 17 21:08:34.806292 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:34.806259    2569 generic.go:356] "Generic (PLEG): container finished" podID="9dc91e48-32c6-42ad-b3e9-7be1ec357d6f" containerID="37ce0bc5c72c3b104e5fe416fc812288ed42ffeffe5c6fc99aa9561af83c4a3d" exitCode=0
Jul 17 21:08:34.806503 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:34.806312    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxjhv" event={"ID":"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f","Type":"ContainerDied","Data":"37ce0bc5c72c3b104e5fe416fc812288ed42ffeffe5c6fc99aa9561af83c4a3d"}
Jul 17 21:08:34.810007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:34.809983    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx" event={"ID":"ba2b267b-b915-414c-9ade-8aa943cfa72e","Type":"ContainerStarted","Data":"0a4bff31837811854e912d8bbc3911c8f04958d042c3cf012baf1d6f673959ec"}
Jul 17 21:08:34.810356 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:34.810328    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:34.810466 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:34.810395    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:34.825713 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:34.825688    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:34.825889 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:34.825875    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:34.847259 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:34.847217    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx" podStartSLOduration=9.168075287 podStartE2EDuration="27.847201566s" totalImagesPullingTime="18.679126279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:34.846235445 +0000 UTC m=+28.605802139" watchObservedRunningTime="2026-07-17 21:08:34.847201566 +0000 UTC m=+28.606768260"
Jul 17 21:08:35.812779 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:35.812751    2569 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 17 21:08:36.205209 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:36.205180    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hnhk2"]
Jul 17 21:08:36.205355 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:36.205330    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:36.205648 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:36.205624    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhk2" podUID="89bdc455-1935-456b-bc58-288a0ed415a0"
Jul 17 21:08:36.208337 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:36.208306    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6bxwh"]
Jul 17 21:08:36.208526 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:36.208510    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:36.208626 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:36.208606    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bxwh" podUID="710a2804-cc1e-49f5-8842-d392bb9aff26"
Jul 17 21:08:36.209015 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:36.208992    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kphjs"]
Jul 17 21:08:36.209091 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:36.209084    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:36.209165 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:36.209150    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kphjs" podUID="3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca"
Jul 17 21:08:36.531120 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:36.531083    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:08:36.817296 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:36.817066    2569 generic.go:356] "Generic (PLEG): container finished" podID="9dc91e48-32c6-42ad-b3e9-7be1ec357d6f" containerID="cd30dd9decf9bf01eb0d3b28884dd2d8e824818ca6bbd3197c596c3179736238" exitCode=0
Jul 17 21:08:36.817296 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:36.817150    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxjhv" event={"ID":"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f","Type":"ContainerDied","Data":"cd30dd9decf9bf01eb0d3b28884dd2d8e824818ca6bbd3197c596c3179736238"}
Jul 17 21:08:37.650730 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:37.650693    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:37.650730 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:37.650720    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:37.650957 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:37.650801    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhk2" podUID="89bdc455-1935-456b-bc58-288a0ed415a0"
Jul 17 21:08:37.650957 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:37.650836    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:37.650957 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:37.650909    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bxwh" podUID="710a2804-cc1e-49f5-8842-d392bb9aff26"
Jul 17 21:08:37.651051 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:37.650979    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kphjs" podUID="3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca"
Jul 17 21:08:38.823847 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:38.823752    2569 generic.go:356] "Generic (PLEG): container finished" podID="9dc91e48-32c6-42ad-b3e9-7be1ec357d6f" containerID="f1c531b148f4fc0ae77b74640bd77023a599acce7828abe939dfbf7bbd2bc846" exitCode=0
Jul 17 21:08:38.823847 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:38.823796    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxjhv" event={"ID":"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f","Type":"ContainerDied","Data":"f1c531b148f4fc0ae77b74640bd77023a599acce7828abe939dfbf7bbd2bc846"}
Jul 17 21:08:39.649863 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:39.649832    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:39.650030 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:39.649839    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:39.650030 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:39.649928    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kphjs" podUID="3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca"
Jul 17 21:08:39.650030 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:39.650021    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hnhk2" podUID="89bdc455-1935-456b-bc58-288a0ed415a0"
Jul 17 21:08:39.650150 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:39.649839    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:39.650150 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:39.650098    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bxwh" podUID="710a2804-cc1e-49f5-8842-d392bb9aff26"
Jul 17 21:08:40.312355 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:40.312319    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4xg\" (UniqueName: \"kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg\") pod \"network-check-target-kphjs\" (UID: \"3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca\") " pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:40.312806 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:40.312511    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:08:40.312806 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:40.312533    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:08:40.312806 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:40.312543    2569 projected.go:196] Error preparing data for projected volume kube-api-access-hs4xg for pod openshift-network-diagnostics/network-check-target-kphjs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:40.312806 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:40.312600    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg podName:3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca nodeName:}" failed. No retries permitted until 2026-07-17 21:09:12.312585826 +0000 UTC m=+66.072152498 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hs4xg" (UniqueName: "kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg") pod "network-check-target-kphjs" (UID: "3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:40.413175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:40.413140    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret\") pod \"global-pull-secret-syncer-6bxwh\" (UID: \"710a2804-cc1e-49f5-8842-d392bb9aff26\") " pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:40.413353 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:40.413198    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs\") pod \"network-metrics-daemon-hnhk2\" (UID: \"89bdc455-1935-456b-bc58-288a0ed415a0\") " pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:40.413353 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:40.413313    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:40.413353 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:40.413313    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:40.413481 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:40.413370    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret podName:710a2804-cc1e-49f5-8842-d392bb9aff26 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:12.413354925 +0000 UTC m=+66.172921601 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret") pod "global-pull-secret-syncer-6bxwh" (UID: "710a2804-cc1e-49f5-8842-d392bb9aff26") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:40.413481 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:40.413385    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs podName:89bdc455-1935-456b-bc58-288a0ed415a0 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:12.413378468 +0000 UTC m=+66.172945139 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs") pod "network-metrics-daemon-hnhk2" (UID: "89bdc455-1935-456b-bc58-288a0ed415a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:41.558820 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.558741    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-117.ec2.internal" event="NodeReady"
Jul 17 21:08:41.559380 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.558895    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 17 21:08:41.593090 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.593053    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn"]
Jul 17 21:08:41.598012 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.597974    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tp2hx"]
Jul 17 21:08:41.598156 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.598095    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn"
Jul 17 21:08:41.600017 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.599994    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tdrl9\""
Jul 17 21:08:41.601002 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.600760    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 17 21:08:41.601002 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.600766    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 17 21:08:41.601002 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.600813    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 17 21:08:41.601002 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.600768    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 17 21:08:41.602007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.601985    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-756f96dd8-2m7s2"]
Jul 17 21:08:41.602119 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.602054    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tp2hx"
Jul 17 21:08:41.604626 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.604462    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:08:41.604626 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.604475    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 17 21:08:41.605754 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.605736    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-jgv9v"]
Jul 17 21:08:41.605885 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.605867    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:41.609371 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.609332    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-tlj4c"]
Jul 17 21:08:41.609488 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.609469    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-jgv9v"
Jul 17 21:08:41.609801 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.609603    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 17 21:08:41.612934 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.612909    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gw6z9"]
Jul 17 21:08:41.613072 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.613053    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-tlj4c"
Jul 17 21:08:41.616497 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.616476    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9szm7"]
Jul 17 21:08:41.616612 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.616599    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gw6z9"
Jul 17 21:08:41.619009 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.618986    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 17 21:08:41.619116 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.618986    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-lxxnw\""
Jul 17 21:08:41.619886 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.619866    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-k7c4s"]
Jul 17 21:08:41.620346 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.620058    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9szm7"
Jul 17 21:08:41.620346 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.620115    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 17 21:08:41.620346 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.620332    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 17 21:08:41.620562 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.619868    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jg7t4\""
Jul 17 21:08:41.620727 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.620709    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 17 21:08:41.620933 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.620916    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cx2ql\""
Jul 17 21:08:41.621313 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.621297    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 17 21:08:41.621621 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.621602    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 17 21:08:41.623705 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.623239    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 17 21:08:41.623957 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.623915    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79fd75b69c-srz29"]
Jul 17 21:08:41.624158 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.624142    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-k7c4s"
Jul 17 21:08:41.624438 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.624420    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 17 21:08:41.627315 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.627296    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn"]
Jul 17 21:08:41.627463 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.627331    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-jgv9v"]
Jul 17 21:08:41.627463 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.627424    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.629200 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.629182    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 17 21:08:41.629303 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.629287    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 17 21:08:41.629484 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.629421    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 17 21:08:41.629738 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.629706    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 17 21:08:41.629825 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.629807    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 17 21:08:41.630055 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.629985    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:08:41.630055 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.630028    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:08:41.630174 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.630164    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 17 21:08:41.630228 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.630209    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hlrp8\""
Jul 17 21:08:41.630960 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.630388    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-r8z62\""
Jul 17 21:08:41.630960 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.630448    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-fjxcp\""
Jul 17 21:08:41.630960 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.630590    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 17 21:08:41.630960 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.630725    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 17 21:08:41.630960 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.630767    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:08:41.630960 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.630872    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-tlj4c"]
Jul 17 21:08:41.650098 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.650040    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:08:41.650669 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.650647    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:08:41.650854 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.650836    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-4lhs5\""
Jul 17 21:08:41.651137 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.651121    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:08:41.651327 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.651306    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 17 21:08:41.651887 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.651557    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 17 21:08:41.651887 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.651644    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 17 21:08:41.651887 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.651722    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 17 21:08:41.651887 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.651756    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:08:41.651887 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.651776    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 17 21:08:41.651887 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.651790    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 17 21:08:41.651887 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.651812    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 17 21:08:41.651887 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.651839    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-97qxw"]
Jul 17 21:08:41.652267 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.652040    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-b28fl\""
Jul 17 21:08:41.656123 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.656099    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 21:08:41.656866 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.656844    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8vkwx\""
Jul 17 21:08:41.657212 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.657193    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 21:08:41.658805 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.658697    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-756f96dd8-2m7s2"]
Jul 17 21:08:41.658965 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.658852    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-97qxw"
Jul 17 21:08:41.659594 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.659566    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79fd75b69c-srz29"]
Jul 17 21:08:41.660901 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.660882    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 21:08:41.662415 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.662382    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-x87k6"]
Jul 17 21:08:41.663847 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.663829    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-szcqz\""
Jul 17 21:08:41.664052 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.664032    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 21:08:41.664229 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.664211    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-j8vps\""
Jul 17 21:08:41.665037 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.664820    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 17 21:08:41.665037 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.665019    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 17 21:08:41.665669 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.665649    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 17 21:08:41.668110 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.668088    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bjqvd"]
Jul 17 21:08:41.668257 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.668231    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-x87k6"
Jul 17 21:08:41.669441 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.669420    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 17 21:08:41.670589 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.670570    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 17 21:08:41.671439 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.671422    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tp2hx"]
Jul 17 21:08:41.671561 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.671547    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9szm7"]
Jul 17 21:08:41.671611 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.671489    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:08:41.672269 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.672249    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-tpwn6\""
Jul 17 21:08:41.674560 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.674542    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rdmx5\""
Jul 17 21:08:41.675772 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.675752    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 17 21:08:41.676341 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.675873    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 17 21:08:41.677709 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.677686    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-k7c4s"]
Jul 17 21:08:41.679102 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.679066    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-97qxw"]
Jul 17 21:08:41.679458 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.679439    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gw6z9"]
Jul 17 21:08:41.680186 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.680167    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-x87k6"]
Jul 17 21:08:41.683050 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.683028    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 17 21:08:41.683256 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.683236    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bjqvd"]
Jul 17 21:08:41.724861 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.724834    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-j5bcn\" (UID: \"a29d8d5f-5087-4cd7-88b3-2d79062600bf\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn"
Jul 17 21:08:41.724977 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.724891    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc4qn\" (UniqueName: \"kubernetes.io/projected/799dbb79-4b48-470c-bcec-14b0bd9c96fd-kube-api-access-pc4qn\") pod \"cluster-samples-operator-79b7c55f87-tlj4c\" (UID: \"799dbb79-4b48-470c-bcec-14b0bd9c96fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-tlj4c"
Jul 17 21:08:41.724977 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.724922    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtbsm\" (UniqueName: \"kubernetes.io/projected/05369579-52bc-4e86-9fe2-32d608748a6d-kube-api-access-mtbsm\") pod \"volume-data-source-validator-78cf4c6f6-gw6z9\" (UID: \"05369579-52bc-4e86-9fe2-32d608748a6d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gw6z9"
Jul 17 21:08:41.724977 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.724951    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6878c658-1dc7-4cd2-811f-ec6a726f07ee-serving-cert\") pod \"insights-operator-555d9d7885-jgv9v\" (UID: \"6878c658-1dc7-4cd2-811f-ec6a726f07ee\") " pod="openshift-insights/insights-operator-555d9d7885-jgv9v"
Jul 17 21:08:41.724977 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.724971    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3287b71-6904-4420-8fbb-65387da8e9f1-config\") pod \"service-ca-operator-5bcbc94c78-tp2hx\" (UID: \"e3287b71-6904-4420-8fbb-65387da8e9f1\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tp2hx"
Jul 17 21:08:41.725179 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.724989    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v98k\" (UniqueName: \"kubernetes.io/projected/f596f331-2b59-4782-b6cc-ff757d3ff94c-kube-api-access-7v98k\") pod \"kube-storage-version-migrator-operator-7658d9b749-9szm7\" (UID: \"f596f331-2b59-4782-b6cc-ff757d3ff94c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9szm7"
Jul 17 21:08:41.725179 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725053    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn6rw\" (UniqueName: \"kubernetes.io/projected/e3287b71-6904-4420-8fbb-65387da8e9f1-kube-api-access-fn6rw\") pod \"service-ca-operator-5bcbc94c78-tp2hx\" (UID: \"e3287b71-6904-4420-8fbb-65387da8e9f1\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tp2hx"
Jul 17 21:08:41.725179 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725090    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-default-certificate\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:41.725179 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725130    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a29d8d5f-5087-4cd7-88b3-2d79062600bf-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-j5bcn\" (UID: \"a29d8d5f-5087-4cd7-88b3-2d79062600bf\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn"
Jul 17 21:08:41.725342 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725180    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6878c658-1dc7-4cd2-811f-ec6a726f07ee-tmp\") pod \"insights-operator-555d9d7885-jgv9v\" (UID: \"6878c658-1dc7-4cd2-811f-ec6a726f07ee\") " pod="openshift-insights/insights-operator-555d9d7885-jgv9v"
Jul 17 21:08:41.725342 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725204    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/43fb78e7-d06f-456a-8e39-b9fe811280aa-image-registry-private-configuration\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.725342 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725225    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/43fb78e7-d06f-456a-8e39-b9fe811280aa-ca-trust-extracted\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.725342 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725295    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0b7933f-7894-4ea9-8cbc-3704888fdeed-serving-cert\") pod \"console-operator-565b7b569d-k7c4s\" (UID: \"c0b7933f-7894-4ea9-8cbc-3704888fdeed\") " pod="openshift-console-operator/console-operator-565b7b569d-k7c4s"
Jul 17 21:08:41.725342 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725327    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-stats-auth\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:41.725531 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725353    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/799dbb79-4b48-470c-bcec-14b0bd9c96fd-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-tlj4c\" (UID: \"799dbb79-4b48-470c-bcec-14b0bd9c96fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-tlj4c"
Jul 17 21:08:41.725531 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725380    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-certificates\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.725531 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725417    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:41.725531 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725479    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6878c658-1dc7-4cd2-811f-ec6a726f07ee-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-jgv9v\" (UID: \"6878c658-1dc7-4cd2-811f-ec6a726f07ee\") " pod="openshift-insights/insights-operator-555d9d7885-jgv9v"
Jul 17 21:08:41.725531 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725517    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcclm\" (UniqueName: \"kubernetes.io/projected/a29d8d5f-5087-4cd7-88b3-2d79062600bf-kube-api-access-fcclm\") pod \"cluster-monitoring-operator-76cf8966f8-j5bcn\" (UID: \"a29d8d5f-5087-4cd7-88b3-2d79062600bf\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn"
Jul 17 21:08:41.725702 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725545    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n56m\" (UniqueName: \"kubernetes.io/projected/c0b7933f-7894-4ea9-8cbc-3704888fdeed-kube-api-access-7n56m\") pod \"console-operator-565b7b569d-k7c4s\" (UID: \"c0b7933f-7894-4ea9-8cbc-3704888fdeed\") " pod="openshift-console-operator/console-operator-565b7b569d-k7c4s"
Jul 17 21:08:41.725702 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725573    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:41.725702 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725599    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0b7933f-7894-4ea9-8cbc-3704888fdeed-config\") pod \"console-operator-565b7b569d-k7c4s\" (UID: \"c0b7933f-7894-4ea9-8cbc-3704888fdeed\") " pod="openshift-console-operator/console-operator-565b7b569d-k7c4s"
Jul 17 21:08:41.725702 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725622    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.725702 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725644    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-bound-sa-token\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.725702 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725671    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgmzw\" (UniqueName: \"kubernetes.io/projected/2cfce8d6-1da4-4003-8183-31744592793b-kube-api-access-fgmzw\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:41.725931 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725712    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6878c658-1dc7-4cd2-811f-ec6a726f07ee-snapshots\") pod \"insights-operator-555d9d7885-jgv9v\" (UID: \"6878c658-1dc7-4cd2-811f-ec6a726f07ee\") " pod="openshift-insights/insights-operator-555d9d7885-jgv9v"
Jul 17 21:08:41.725931 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725735    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f596f331-2b59-4782-b6cc-ff757d3ff94c-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-9szm7\" (UID: \"f596f331-2b59-4782-b6cc-ff757d3ff94c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9szm7"
Jul 17 21:08:41.725931 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725765    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm2qj\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-kube-api-access-cm2qj\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.725931 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725805    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbbvg\" (UniqueName: \"kubernetes.io/projected/6878c658-1dc7-4cd2-811f-ec6a726f07ee-kube-api-access-bbbvg\") pod \"insights-operator-555d9d7885-jgv9v\" (UID: \"6878c658-1dc7-4cd2-811f-ec6a726f07ee\") " pod="openshift-insights/insights-operator-555d9d7885-jgv9v"
Jul 17 21:08:41.725931 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725852    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f596f331-2b59-4782-b6cc-ff757d3ff94c-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-9szm7\" (UID: \"f596f331-2b59-4782-b6cc-ff757d3ff94c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9szm7"
Jul 17 21:08:41.725931 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725884    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/43fb78e7-d06f-456a-8e39-b9fe811280aa-installation-pull-secrets\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.725931 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725910    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6878c658-1dc7-4cd2-811f-ec6a726f07ee-service-ca-bundle\") pod \"insights-operator-555d9d7885-jgv9v\" (UID: \"6878c658-1dc7-4cd2-811f-ec6a726f07ee\") " pod="openshift-insights/insights-operator-555d9d7885-jgv9v"
Jul 17 21:08:41.726343 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725933    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3287b71-6904-4420-8fbb-65387da8e9f1-serving-cert\") pod \"service-ca-operator-5bcbc94c78-tp2hx\" (UID: \"e3287b71-6904-4420-8fbb-65387da8e9f1\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tp2hx"
Jul 17 21:08:41.726343 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.725968    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43fb78e7-d06f-456a-8e39-b9fe811280aa-trusted-ca\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.726343 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.726003    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0b7933f-7894-4ea9-8cbc-3704888fdeed-trusted-ca\") pod \"console-operator-565b7b569d-k7c4s\" (UID: \"c0b7933f-7894-4ea9-8cbc-3704888fdeed\") " pod="openshift-console-operator/console-operator-565b7b569d-k7c4s"
Jul 17 21:08:41.799240 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.799201    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:41.826666 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.826591    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8859418c-d1a7-4728-989b-04addc542331-config-volume\") pod \"dns-default-bjqvd\" (UID: \"8859418c-d1a7-4728-989b-04addc542331\") " pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:08:41.826666 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.826620    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8859418c-d1a7-4728-989b-04addc542331-tmp-dir\") pod \"dns-default-bjqvd\" (UID: \"8859418c-d1a7-4728-989b-04addc542331\") " pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:08:41.826666 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.826642    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-stats-auth\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:41.826666 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.826655    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ca96db1-3372-483e-9e19-2fb5046e81bc-cert\") pod \"ingress-canary-97qxw\" (UID: \"3ca96db1-3372-483e-9e19-2fb5046e81bc\") " pod="openshift-ingress-canary/ingress-canary-97qxw"
Jul 17 21:08:41.826980 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.826672    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/799dbb79-4b48-470c-bcec-14b0bd9c96fd-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-tlj4c\" (UID: \"799dbb79-4b48-470c-bcec-14b0bd9c96fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-tlj4c"
Jul 17 21:08:41.826980 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.826711    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-certificates\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.826980 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.826803    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:41.826980 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:41.826847    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 21:08:41.826980 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:41.826917    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799dbb79-4b48-470c-bcec-14b0bd9c96fd-samples-operator-tls podName:799dbb79-4b48-470c-bcec-14b0bd9c96fd nodeName:}" failed. No retries permitted until 2026-07-17 21:08:42.326895644 +0000 UTC m=+36.086462322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/799dbb79-4b48-470c-bcec-14b0bd9c96fd-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-tlj4c" (UID: "799dbb79-4b48-470c-bcec-14b0bd9c96fd") : secret "samples-operator-tls" not found
Jul 17 21:08:41.826980 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.826844    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6878c658-1dc7-4cd2-811f-ec6a726f07ee-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-jgv9v\" (UID: \"6878c658-1dc7-4cd2-811f-ec6a726f07ee\") " pod="openshift-insights/insights-operator-555d9d7885-jgv9v"
Jul 17 21:08:41.827318 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:41.826991    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle podName:2cfce8d6-1da4-4003-8183-31744592793b nodeName:}" failed. No retries permitted until 2026-07-17 21:08:42.32696916 +0000 UTC m=+36.086535845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle") pod "router-default-756f96dd8-2m7s2" (UID: "2cfce8d6-1da4-4003-8183-31744592793b") : configmap references non-existent config key: service-ca.crt
Jul 17 21:08:41.827318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827050    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcclm\" (UniqueName: \"kubernetes.io/projected/a29d8d5f-5087-4cd7-88b3-2d79062600bf-kube-api-access-fcclm\") pod \"cluster-monitoring-operator-76cf8966f8-j5bcn\" (UID: \"a29d8d5f-5087-4cd7-88b3-2d79062600bf\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn"
Jul 17 21:08:41.827318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827079    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n56m\" (UniqueName: \"kubernetes.io/projected/c0b7933f-7894-4ea9-8cbc-3704888fdeed-kube-api-access-7n56m\") pod \"console-operator-565b7b569d-k7c4s\" (UID: \"c0b7933f-7894-4ea9-8cbc-3704888fdeed\") " pod="openshift-console-operator/console-operator-565b7b569d-k7c4s"
Jul 17 21:08:41.827318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827107    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:41.827318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827136    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0b7933f-7894-4ea9-8cbc-3704888fdeed-config\") pod \"console-operator-565b7b569d-k7c4s\" (UID: \"c0b7933f-7894-4ea9-8cbc-3704888fdeed\") " pod="openshift-console-operator/console-operator-565b7b569d-k7c4s"
Jul 17 21:08:41.827318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827158    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.827318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827184    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-bound-sa-token\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.827318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827205    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgmzw\" (UniqueName: \"kubernetes.io/projected/2cfce8d6-1da4-4003-8183-31744592793b-kube-api-access-fgmzw\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:41.827318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827240    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6878c658-1dc7-4cd2-811f-ec6a726f07ee-snapshots\") pod \"insights-operator-555d9d7885-jgv9v\" (UID: \"6878c658-1dc7-4cd2-811f-ec6a726f07ee\") " pod="openshift-insights/insights-operator-555d9d7885-jgv9v"
Jul 17 21:08:41.827318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827262    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f596f331-2b59-4782-b6cc-ff757d3ff94c-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-9szm7\" (UID: \"f596f331-2b59-4782-b6cc-ff757d3ff94c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9szm7"
Jul 17 21:08:41.827318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827293    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm2qj\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-kube-api-access-cm2qj\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.827318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827320    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbbvg\" (UniqueName: \"kubernetes.io/projected/6878c658-1dc7-4cd2-811f-ec6a726f07ee-kube-api-access-bbbvg\") pod \"insights-operator-555d9d7885-jgv9v\" (UID: \"6878c658-1dc7-4cd2-811f-ec6a726f07ee\") " pod="openshift-insights/insights-operator-555d9d7885-jgv9v"
Jul 17 21:08:41.827964 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827343    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f596f331-2b59-4782-b6cc-ff757d3ff94c-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-9szm7\" (UID: \"f596f331-2b59-4782-b6cc-ff757d3ff94c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9szm7"
Jul 17 21:08:41.827964 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827342    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-certificates\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.827964 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827368    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x679q\" (UniqueName: \"kubernetes.io/projected/9fd85725-3ea0-4c51-aa5f-3238c4ba6acd-kube-api-access-x679q\") pod \"network-check-source-7bc5b6bf85-x87k6\" (UID: \"9fd85725-3ea0-4c51-aa5f-3238c4ba6acd\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-x87k6"
Jul 17 21:08:41.827964 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:41.827482    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 17 21:08:41.827964 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:41.827500    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:08:41.827964 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:41.827515    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79fd75b69c-srz29: secret "image-registry-tls" not found
Jul 17 21:08:41.827964 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:41.827528    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs podName:2cfce8d6-1da4-4003-8183-31744592793b nodeName:}" failed. No retries permitted until 2026-07-17 21:08:42.327514292 +0000 UTC m=+36.087080979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs") pod "router-default-756f96dd8-2m7s2" (UID: "2cfce8d6-1da4-4003-8183-31744592793b") : secret "router-metrics-certs-default" not found
Jul 17 21:08:41.827964 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:41.827578    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls podName:43fb78e7-d06f-456a-8e39-b9fe811280aa nodeName:}" failed. No retries permitted until 2026-07-17 21:08:42.327552223 +0000 UTC m=+36.087118915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls") pod "image-registry-79fd75b69c-srz29" (UID: "43fb78e7-d06f-456a-8e39-b9fe811280aa") : secret "image-registry-tls" not found
Jul 17 21:08:41.827964 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827605    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/43fb78e7-d06f-456a-8e39-b9fe811280aa-installation-pull-secrets\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.827964 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827639    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6878c658-1dc7-4cd2-811f-ec6a726f07ee-service-ca-bundle\") pod \"insights-operator-555d9d7885-jgv9v\" (UID: \"6878c658-1dc7-4cd2-811f-ec6a726f07ee\") " pod="openshift-insights/insights-operator-555d9d7885-jgv9v"
Jul 17 21:08:41.827964 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827676    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3287b71-6904-4420-8fbb-65387da8e9f1-serving-cert\") pod \"service-ca-operator-5bcbc94c78-tp2hx\" (UID: \"e3287b71-6904-4420-8fbb-65387da8e9f1\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tp2hx"
Jul 17 21:08:41.827964 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827707    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43fb78e7-d06f-456a-8e39-b9fe811280aa-trusted-ca\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.827964 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827742    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0b7933f-7894-4ea9-8cbc-3704888fdeed-trusted-ca\") pod \"console-operator-565b7b569d-k7c4s\" (UID: \"c0b7933f-7894-4ea9-8cbc-3704888fdeed\") " pod="openshift-console-operator/console-operator-565b7b569d-k7c4s"
Jul 17 21:08:41.828612 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827765    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xqt5\" (UniqueName: \"kubernetes.io/projected/3ca96db1-3372-483e-9e19-2fb5046e81bc-kube-api-access-4xqt5\") pod \"ingress-canary-97qxw\" (UID: \"3ca96db1-3372-483e-9e19-2fb5046e81bc\") " pod="openshift-ingress-canary/ingress-canary-97qxw"
Jul 17 21:08:41.828612 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827791    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls\") pod \"dns-default-bjqvd\" (UID: \"8859418c-d1a7-4728-989b-04addc542331\") " pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:08:41.828612 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827839    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6878c658-1dc7-4cd2-811f-ec6a726f07ee-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-jgv9v\" (UID: \"6878c658-1dc7-4cd2-811f-ec6a726f07ee\") " pod="openshift-insights/insights-operator-555d9d7885-jgv9v"
Jul 17 21:08:41.828612 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827848    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-j5bcn\" (UID: \"a29d8d5f-5087-4cd7-88b3-2d79062600bf\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn"
Jul 17 21:08:41.828612 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827890    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc4qn\" (UniqueName: \"kubernetes.io/projected/799dbb79-4b48-470c-bcec-14b0bd9c96fd-kube-api-access-pc4qn\") pod \"cluster-samples-operator-79b7c55f87-tlj4c\" (UID: \"799dbb79-4b48-470c-bcec-14b0bd9c96fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-tlj4c"
Jul 17 21:08:41.828612 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827917    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtbsm\" (UniqueName: \"kubernetes.io/projected/05369579-52bc-4e86-9fe2-32d608748a6d-kube-api-access-mtbsm\") pod \"volume-data-source-validator-78cf4c6f6-gw6z9\" (UID: \"05369579-52bc-4e86-9fe2-32d608748a6d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gw6z9"
Jul 17 21:08:41.828612 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827946    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5snq\" (UniqueName: \"kubernetes.io/projected/8859418c-d1a7-4728-989b-04addc542331-kube-api-access-l5snq\") pod \"dns-default-bjqvd\" (UID: \"8859418c-d1a7-4728-989b-04addc542331\") " pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:08:41.828612 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.827988    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6878c658-1dc7-4cd2-811f-ec6a726f07ee-serving-cert\") pod \"insights-operator-555d9d7885-jgv9v\" (UID: \"6878c658-1dc7-4cd2-811f-ec6a726f07ee\") " pod="openshift-insights/insights-operator-555d9d7885-jgv9v"
Jul 17 21:08:41.828612 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.828006    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6878c658-1dc7-4cd2-811f-ec6a726f07ee-snapshots\") pod \"insights-operator-555d9d7885-jgv9v\" (UID: \"6878c658-1dc7-4cd2-811f-ec6a726f07ee\") " pod="openshift-insights/insights-operator-555d9d7885-jgv9v"
Jul 17 21:08:41.828612 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.828011    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3287b71-6904-4420-8fbb-65387da8e9f1-config\") pod \"service-ca-operator-5bcbc94c78-tp2hx\" (UID: \"e3287b71-6904-4420-8fbb-65387da8e9f1\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tp2hx"
Jul 17 21:08:41.828612 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.828076    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v98k\" (UniqueName: \"kubernetes.io/projected/f596f331-2b59-4782-b6cc-ff757d3ff94c-kube-api-access-7v98k\") pod \"kube-storage-version-migrator-operator-7658d9b749-9szm7\" (UID: \"f596f331-2b59-4782-b6cc-ff757d3ff94c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9szm7"
Jul 17 21:08:41.828612 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.828107    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn6rw\" (UniqueName: \"kubernetes.io/projected/e3287b71-6904-4420-8fbb-65387da8e9f1-kube-api-access-fn6rw\") pod \"service-ca-operator-5bcbc94c78-tp2hx\" (UID: \"e3287b71-6904-4420-8fbb-65387da8e9f1\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tp2hx"
Jul 17 21:08:41.828612 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.828139    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-default-certificate\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:41.829293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.828169    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a29d8d5f-5087-4cd7-88b3-2d79062600bf-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-j5bcn\" (UID: \"a29d8d5f-5087-4cd7-88b3-2d79062600bf\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn"
Jul 17 21:08:41.829293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.828194    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0b7933f-7894-4ea9-8cbc-3704888fdeed-config\") pod \"console-operator-565b7b569d-k7c4s\" (UID: \"c0b7933f-7894-4ea9-8cbc-3704888fdeed\") " pod="openshift-console-operator/console-operator-565b7b569d-k7c4s"
Jul 17 21:08:41.829293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.828196    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6878c658-1dc7-4cd2-811f-ec6a726f07ee-tmp\") pod \"insights-operator-555d9d7885-jgv9v\" (UID: \"6878c658-1dc7-4cd2-811f-ec6a726f07ee\") " pod="openshift-insights/insights-operator-555d9d7885-jgv9v"
Jul 17 21:08:41.829293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.828238    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/43fb78e7-d06f-456a-8e39-b9fe811280aa-image-registry-private-configuration\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.829293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.828279    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6878c658-1dc7-4cd2-811f-ec6a726f07ee-service-ca-bundle\") pod \"insights-operator-555d9d7885-jgv9v\" (UID: \"6878c658-1dc7-4cd2-811f-ec6a726f07ee\") " pod="openshift-insights/insights-operator-555d9d7885-jgv9v"
Jul 17 21:08:41.829293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.828261    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/43fb78e7-d06f-456a-8e39-b9fe811280aa-ca-trust-extracted\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.829293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.828350    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0b7933f-7894-4ea9-8cbc-3704888fdeed-serving-cert\") pod \"console-operator-565b7b569d-k7c4s\" (UID: \"c0b7933f-7894-4ea9-8cbc-3704888fdeed\") " pod="openshift-console-operator/console-operator-565b7b569d-k7c4s"
Jul 17 21:08:41.829293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.828456    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6878c658-1dc7-4cd2-811f-ec6a726f07ee-tmp\") pod \"insights-operator-555d9d7885-jgv9v\" (UID: \"6878c658-1dc7-4cd2-811f-ec6a726f07ee\") " pod="openshift-insights/insights-operator-555d9d7885-jgv9v"
Jul 17 21:08:41.829293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.828480    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f596f331-2b59-4782-b6cc-ff757d3ff94c-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-9szm7\" (UID: \"f596f331-2b59-4782-b6cc-ff757d3ff94c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9szm7"
Jul 17 21:08:41.829293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.828535    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/43fb78e7-d06f-456a-8e39-b9fe811280aa-ca-trust-extracted\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.829293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.828651    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0b7933f-7894-4ea9-8cbc-3704888fdeed-trusted-ca\") pod \"console-operator-565b7b569d-k7c4s\" (UID: \"c0b7933f-7894-4ea9-8cbc-3704888fdeed\") " pod="openshift-console-operator/console-operator-565b7b569d-k7c4s"
Jul 17 21:08:41.829293 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:41.829044    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 21:08:41.829293 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:41.829090    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls podName:a29d8d5f-5087-4cd7-88b3-2d79062600bf nodeName:}" failed. No retries permitted until 2026-07-17 21:08:42.32907562 +0000 UTC m=+36.088642302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-j5bcn" (UID: "a29d8d5f-5087-4cd7-88b3-2d79062600bf") : secret "cluster-monitoring-operator-tls" not found
Jul 17 21:08:41.830033 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.829503    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43fb78e7-d06f-456a-8e39-b9fe811280aa-trusted-ca\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.830033 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.829646    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3287b71-6904-4420-8fbb-65387da8e9f1-config\") pod \"service-ca-operator-5bcbc94c78-tp2hx\" (UID: \"e3287b71-6904-4420-8fbb-65387da8e9f1\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tp2hx"
Jul 17 21:08:41.830164 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.830138    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a29d8d5f-5087-4cd7-88b3-2d79062600bf-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-j5bcn\" (UID: \"a29d8d5f-5087-4cd7-88b3-2d79062600bf\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn"
Jul 17 21:08:41.832722 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.832677    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-stats-auth\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:41.832842 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.832811    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/43fb78e7-d06f-456a-8e39-b9fe811280aa-installation-pull-secrets\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.833116 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.833095    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f596f331-2b59-4782-b6cc-ff757d3ff94c-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-9szm7\" (UID: \"f596f331-2b59-4782-b6cc-ff757d3ff94c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9szm7"
Jul 17 21:08:41.833116 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.833107    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3287b71-6904-4420-8fbb-65387da8e9f1-serving-cert\") pod \"service-ca-operator-5bcbc94c78-tp2hx\" (UID: \"e3287b71-6904-4420-8fbb-65387da8e9f1\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tp2hx"
Jul 17 21:08:41.833637 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.833596    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-default-certificate\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:41.833637 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.833606    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0b7933f-7894-4ea9-8cbc-3704888fdeed-serving-cert\") pod \"console-operator-565b7b569d-k7c4s\" (UID: \"c0b7933f-7894-4ea9-8cbc-3704888fdeed\") " pod="openshift-console-operator/console-operator-565b7b569d-k7c4s"
Jul 17 21:08:41.834622 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.834584    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/43fb78e7-d06f-456a-8e39-b9fe811280aa-image-registry-private-configuration\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.834715 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.834660    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6878c658-1dc7-4cd2-811f-ec6a726f07ee-serving-cert\") pod \"insights-operator-555d9d7885-jgv9v\" (UID: \"6878c658-1dc7-4cd2-811f-ec6a726f07ee\") " pod="openshift-insights/insights-operator-555d9d7885-jgv9v"
Jul 17 21:08:41.839940 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.839765    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc4qn\" (UniqueName: \"kubernetes.io/projected/799dbb79-4b48-470c-bcec-14b0bd9c96fd-kube-api-access-pc4qn\") pod \"cluster-samples-operator-79b7c55f87-tlj4c\" (UID: \"799dbb79-4b48-470c-bcec-14b0bd9c96fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-tlj4c"
Jul 17 21:08:41.842663 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.842607    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n56m\" (UniqueName: \"kubernetes.io/projected/c0b7933f-7894-4ea9-8cbc-3704888fdeed-kube-api-access-7n56m\") pod \"console-operator-565b7b569d-k7c4s\" (UID: \"c0b7933f-7894-4ea9-8cbc-3704888fdeed\") " pod="openshift-console-operator/console-operator-565b7b569d-k7c4s"
Jul 17 21:08:41.842855 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.842831    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgmzw\" (UniqueName: \"kubernetes.io/projected/2cfce8d6-1da4-4003-8183-31744592793b-kube-api-access-fgmzw\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:41.843607 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.843564    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcclm\" (UniqueName: \"kubernetes.io/projected/a29d8d5f-5087-4cd7-88b3-2d79062600bf-kube-api-access-fcclm\") pod \"cluster-monitoring-operator-76cf8966f8-j5bcn\" (UID: \"a29d8d5f-5087-4cd7-88b3-2d79062600bf\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn"
Jul 17 21:08:41.843959 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.843891    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn6rw\" (UniqueName: \"kubernetes.io/projected/e3287b71-6904-4420-8fbb-65387da8e9f1-kube-api-access-fn6rw\") pod \"service-ca-operator-5bcbc94c78-tp2hx\" (UID: \"e3287b71-6904-4420-8fbb-65387da8e9f1\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tp2hx"
Jul 17 21:08:41.845427 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.844729    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtbsm\" (UniqueName: \"kubernetes.io/projected/05369579-52bc-4e86-9fe2-32d608748a6d-kube-api-access-mtbsm\") pod \"volume-data-source-validator-78cf4c6f6-gw6z9\" (UID: \"05369579-52bc-4e86-9fe2-32d608748a6d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gw6z9"
Jul 17 21:08:41.845427 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.845140    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-bound-sa-token\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.845427 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.845249    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbbvg\" (UniqueName: \"kubernetes.io/projected/6878c658-1dc7-4cd2-811f-ec6a726f07ee-kube-api-access-bbbvg\") pod \"insights-operator-555d9d7885-jgv9v\" (UID: \"6878c658-1dc7-4cd2-811f-ec6a726f07ee\") " pod="openshift-insights/insights-operator-555d9d7885-jgv9v"
Jul 17 21:08:41.845845 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.845821    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v98k\" (UniqueName: \"kubernetes.io/projected/f596f331-2b59-4782-b6cc-ff757d3ff94c-kube-api-access-7v98k\") pod \"kube-storage-version-migrator-operator-7658d9b749-9szm7\" (UID: \"f596f331-2b59-4782-b6cc-ff757d3ff94c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9szm7"
Jul 17 21:08:41.846520 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.846500    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm2qj\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-kube-api-access-cm2qj\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:41.902619 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.902578    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:41.918485 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.918454    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tp2hx"
Jul 17 21:08:41.929248 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.929218    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xqt5\" (UniqueName: \"kubernetes.io/projected/3ca96db1-3372-483e-9e19-2fb5046e81bc-kube-api-access-4xqt5\") pod \"ingress-canary-97qxw\" (UID: \"3ca96db1-3372-483e-9e19-2fb5046e81bc\") " pod="openshift-ingress-canary/ingress-canary-97qxw"
Jul 17 21:08:41.929364 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.929261    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls\") pod \"dns-default-bjqvd\" (UID: \"8859418c-d1a7-4728-989b-04addc542331\") " pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:08:41.929479 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:41.929458    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:08:41.929545 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:41.929528    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls podName:8859418c-d1a7-4728-989b-04addc542331 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:42.429507801 +0000 UTC m=+36.189074705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls") pod "dns-default-bjqvd" (UID: "8859418c-d1a7-4728-989b-04addc542331") : secret "dns-default-metrics-tls" not found
Jul 17 21:08:41.929604 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.929579    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5snq\" (UniqueName: \"kubernetes.io/projected/8859418c-d1a7-4728-989b-04addc542331-kube-api-access-l5snq\") pod \"dns-default-bjqvd\" (UID: \"8859418c-d1a7-4728-989b-04addc542331\") " pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:08:41.929656 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.929632    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8859418c-d1a7-4728-989b-04addc542331-config-volume\") pod \"dns-default-bjqvd\" (UID: \"8859418c-d1a7-4728-989b-04addc542331\") " pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:08:41.929700 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.929657    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8859418c-d1a7-4728-989b-04addc542331-tmp-dir\") pod \"dns-default-bjqvd\" (UID: \"8859418c-d1a7-4728-989b-04addc542331\") " pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:08:41.929700 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.929684    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ca96db1-3372-483e-9e19-2fb5046e81bc-cert\") pod \"ingress-canary-97qxw\" (UID: \"3ca96db1-3372-483e-9e19-2fb5046e81bc\") " pod="openshift-ingress-canary/ingress-canary-97qxw"
Jul 17 21:08:41.929861 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.929802    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x679q\" (UniqueName: \"kubernetes.io/projected/9fd85725-3ea0-4c51-aa5f-3238c4ba6acd-kube-api-access-x679q\") pod \"network-check-source-7bc5b6bf85-x87k6\" (UID: \"9fd85725-3ea0-4c51-aa5f-3238c4ba6acd\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-x87k6"
Jul 17 21:08:41.930156 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.930129    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8859418c-d1a7-4728-989b-04addc542331-tmp-dir\") pod \"dns-default-bjqvd\" (UID: \"8859418c-d1a7-4728-989b-04addc542331\") " pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:08:41.930321 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.930301    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8859418c-d1a7-4728-989b-04addc542331-config-volume\") pod \"dns-default-bjqvd\" (UID: \"8859418c-d1a7-4728-989b-04addc542331\") " pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:08:41.932846 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.932635    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ca96db1-3372-483e-9e19-2fb5046e81bc-cert\") pod \"ingress-canary-97qxw\" (UID: \"3ca96db1-3372-483e-9e19-2fb5046e81bc\") " pod="openshift-ingress-canary/ingress-canary-97qxw"
Jul 17 21:08:41.935154 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.935133    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-jgv9v"
Jul 17 21:08:41.938775 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.938716    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5snq\" (UniqueName: \"kubernetes.io/projected/8859418c-d1a7-4728-989b-04addc542331-kube-api-access-l5snq\") pod \"dns-default-bjqvd\" (UID: \"8859418c-d1a7-4728-989b-04addc542331\") " pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:08:41.939577 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.939111    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x679q\" (UniqueName: \"kubernetes.io/projected/9fd85725-3ea0-4c51-aa5f-3238c4ba6acd-kube-api-access-x679q\") pod \"network-check-source-7bc5b6bf85-x87k6\" (UID: \"9fd85725-3ea0-4c51-aa5f-3238c4ba6acd\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-x87k6"
Jul 17 21:08:41.941053 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.941029    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xqt5\" (UniqueName: \"kubernetes.io/projected/3ca96db1-3372-483e-9e19-2fb5046e81bc-kube-api-access-4xqt5\") pod \"ingress-canary-97qxw\" (UID: \"3ca96db1-3372-483e-9e19-2fb5046e81bc\") " pod="openshift-ingress-canary/ingress-canary-97qxw"
Jul 17 21:08:41.961556 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.961271    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gw6z9"
Jul 17 21:08:41.966763 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.966700    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9szm7"
Jul 17 21:08:41.973242 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:41.973191    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-k7c4s"
Jul 17 21:08:42.013204 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.012915    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-97qxw"
Jul 17 21:08:42.019255 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.019061    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-x87k6"
Jul 17 21:08:42.169997 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.169864    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tp2hx"]
Jul 17 21:08:42.175880 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:42.175844    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3287b71_6904_4420_8fbb_65387da8e9f1.slice/crio-e2a6d29b2097230a9b827df08a83a67b5d01c0344de6bc14de66fb45dea2b82a WatchSource:0}: Error finding container e2a6d29b2097230a9b827df08a83a67b5d01c0344de6bc14de66fb45dea2b82a: Status 404 returned error can't find the container with id e2a6d29b2097230a9b827df08a83a67b5d01c0344de6bc14de66fb45dea2b82a
Jul 17 21:08:42.185319 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.185248    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-jgv9v"]
Jul 17 21:08:42.191965 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:42.190918    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6878c658_1dc7_4cd2_811f_ec6a726f07ee.slice/crio-dec7fd6c593772e0581f3b7b9f6ace9992e2c3fb55c77ab6ab0fd11d81f081ef WatchSource:0}: Error finding container dec7fd6c593772e0581f3b7b9f6ace9992e2c3fb55c77ab6ab0fd11d81f081ef: Status 404 returned error can't find the container with id dec7fd6c593772e0581f3b7b9f6ace9992e2c3fb55c77ab6ab0fd11d81f081ef
Jul 17 21:08:42.200052 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.199778    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gw6z9"]
Jul 17 21:08:42.208952 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.208883    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9szm7"]
Jul 17 21:08:42.210867 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:42.210833    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf596f331_2b59_4782_b6cc_ff757d3ff94c.slice/crio-713c80488a805614ceecb1ca20bfc4ff0be1558ca9518da77a7cc736152a1dee WatchSource:0}: Error finding container 713c80488a805614ceecb1ca20bfc4ff0be1558ca9518da77a7cc736152a1dee: Status 404 returned error can't find the container with id 713c80488a805614ceecb1ca20bfc4ff0be1558ca9518da77a7cc736152a1dee
Jul 17 21:08:42.211209 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:42.211180    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05369579_52bc_4e86_9fe2_32d608748a6d.slice/crio-fa7650058f779699812153fc8a6fedf293953746fad3142ef83cb65c4ec819f3 WatchSource:0}: Error finding container fa7650058f779699812153fc8a6fedf293953746fad3142ef83cb65c4ec819f3: Status 404 returned error can't find the container with id fa7650058f779699812153fc8a6fedf293953746fad3142ef83cb65c4ec819f3
Jul 17 21:08:42.215112 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.215074    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-k7c4s"]
Jul 17 21:08:42.219170 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:42.219119    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b7933f_7894_4ea9_8cbc_3704888fdeed.slice/crio-ffad332f96ade52caa4d21609faae3b6c896aa4bb21d7feea3216adcefa6e705 WatchSource:0}: Error finding container ffad332f96ade52caa4d21609faae3b6c896aa4bb21d7feea3216adcefa6e705: Status 404 returned error can't find the container with id ffad332f96ade52caa4d21609faae3b6c896aa4bb21d7feea3216adcefa6e705
Jul 17 21:08:42.249085 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.249055    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-97qxw"]
Jul 17 21:08:42.252505 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:42.252465    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca96db1_3372_483e_9e19_2fb5046e81bc.slice/crio-5df75551ca8c5e96d99cfde58c680371697c2c3fcc6cead7685b739686719ed9 WatchSource:0}: Error finding container 5df75551ca8c5e96d99cfde58c680371697c2c3fcc6cead7685b739686719ed9: Status 404 returned error can't find the container with id 5df75551ca8c5e96d99cfde58c680371697c2c3fcc6cead7685b739686719ed9
Jul 17 21:08:42.280575 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.280543    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-x87k6"]
Jul 17 21:08:42.283549 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:42.283521    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd85725_3ea0_4c51_aa5f_3238c4ba6acd.slice/crio-1e28e289ea19c38d285339807108b1193ba096d03f604e65dc5aa600996747a1 WatchSource:0}: Error finding container 1e28e289ea19c38d285339807108b1193ba096d03f604e65dc5aa600996747a1: Status 404 returned error can't find the container with id 1e28e289ea19c38d285339807108b1193ba096d03f604e65dc5aa600996747a1
Jul 17 21:08:42.335303 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.335226    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-j5bcn\" (UID: \"a29d8d5f-5087-4cd7-88b3-2d79062600bf\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn"
Jul 17 21:08:42.335303 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.335294    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/799dbb79-4b48-470c-bcec-14b0bd9c96fd-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-tlj4c\" (UID: \"799dbb79-4b48-470c-bcec-14b0bd9c96fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-tlj4c"
Jul 17 21:08:42.335586 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.335323    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:42.335586 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.335356    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:42.335586 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:42.335382    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 21:08:42.335586 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.335436    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:42.335586 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:42.335455    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 17 21:08:42.335586 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:42.335472    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls podName:a29d8d5f-5087-4cd7-88b3-2d79062600bf nodeName:}" failed. No retries permitted until 2026-07-17 21:08:43.335452845 +0000 UTC m=+37.095019531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-j5bcn" (UID: "a29d8d5f-5087-4cd7-88b3-2d79062600bf") : secret "cluster-monitoring-operator-tls" not found
Jul 17 21:08:42.335586 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:42.335493    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs podName:2cfce8d6-1da4-4003-8183-31744592793b nodeName:}" failed. No retries permitted until 2026-07-17 21:08:43.335483432 +0000 UTC m=+37.095050108 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs") pod "router-default-756f96dd8-2m7s2" (UID: "2cfce8d6-1da4-4003-8183-31744592793b") : secret "router-metrics-certs-default" not found
Jul 17 21:08:42.335586 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:42.335502    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 21:08:42.335586 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:42.335514    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:08:42.335586 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:42.335529    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79fd75b69c-srz29: secret "image-registry-tls" not found
Jul 17 21:08:42.335586 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:42.335509    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle podName:2cfce8d6-1da4-4003-8183-31744592793b nodeName:}" failed. No retries permitted until 2026-07-17 21:08:43.3355017 +0000 UTC m=+37.095068371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle") pod "router-default-756f96dd8-2m7s2" (UID: "2cfce8d6-1da4-4003-8183-31744592793b") : configmap references non-existent config key: service-ca.crt
Jul 17 21:08:42.335586 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:42.335571    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799dbb79-4b48-470c-bcec-14b0bd9c96fd-samples-operator-tls podName:799dbb79-4b48-470c-bcec-14b0bd9c96fd nodeName:}" failed. No retries permitted until 2026-07-17 21:08:43.335553364 +0000 UTC m=+37.095120047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/799dbb79-4b48-470c-bcec-14b0bd9c96fd-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-tlj4c" (UID: "799dbb79-4b48-470c-bcec-14b0bd9c96fd") : secret "samples-operator-tls" not found
Jul 17 21:08:42.335586 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:42.335592    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls podName:43fb78e7-d06f-456a-8e39-b9fe811280aa nodeName:}" failed. No retries permitted until 2026-07-17 21:08:43.335581855 +0000 UTC m=+37.095148537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls") pod "image-registry-79fd75b69c-srz29" (UID: "43fb78e7-d06f-456a-8e39-b9fe811280aa") : secret "image-registry-tls" not found
Jul 17 21:08:42.440046 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.439980    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls\") pod \"dns-default-bjqvd\" (UID: \"8859418c-d1a7-4728-989b-04addc542331\") " pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:08:42.440224 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:42.440135    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:08:42.440611 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:42.440590    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls podName:8859418c-d1a7-4728-989b-04addc542331 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:43.440570062 +0000 UTC m=+37.200136746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls") pod "dns-default-bjqvd" (UID: "8859418c-d1a7-4728-989b-04addc542331") : secret "dns-default-metrics-tls" not found
Jul 17 21:08:42.784749 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.784559    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:42.836933 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.836847    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9szm7" event={"ID":"f596f331-2b59-4782-b6cc-ff757d3ff94c","Type":"ContainerStarted","Data":"713c80488a805614ceecb1ca20bfc4ff0be1558ca9518da77a7cc736152a1dee"}
Jul 17 21:08:42.838718 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.838649    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gw6z9" event={"ID":"05369579-52bc-4e86-9fe2-32d608748a6d","Type":"ContainerStarted","Data":"fa7650058f779699812153fc8a6fedf293953746fad3142ef83cb65c4ec819f3"}
Jul 17 21:08:42.840351 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.840322    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-jgv9v" event={"ID":"6878c658-1dc7-4cd2-811f-ec6a726f07ee","Type":"ContainerStarted","Data":"dec7fd6c593772e0581f3b7b9f6ace9992e2c3fb55c77ab6ab0fd11d81f081ef"}
Jul 17 21:08:42.841995 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.841893    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-k7c4s" event={"ID":"c0b7933f-7894-4ea9-8cbc-3704888fdeed","Type":"ContainerStarted","Data":"ffad332f96ade52caa4d21609faae3b6c896aa4bb21d7feea3216adcefa6e705"}
Jul 17 21:08:42.844092 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.844038    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-97qxw" event={"ID":"3ca96db1-3372-483e-9e19-2fb5046e81bc","Type":"ContainerStarted","Data":"5df75551ca8c5e96d99cfde58c680371697c2c3fcc6cead7685b739686719ed9"}
Jul 17 21:08:42.846684 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.846637    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-x87k6" event={"ID":"9fd85725-3ea0-4c51-aa5f-3238c4ba6acd","Type":"ContainerStarted","Data":"1e28e289ea19c38d285339807108b1193ba096d03f604e65dc5aa600996747a1"}
Jul 17 21:08:42.847832 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.847810    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tp2hx" event={"ID":"e3287b71-6904-4420-8fbb-65387da8e9f1","Type":"ContainerStarted","Data":"e2a6d29b2097230a9b827df08a83a67b5d01c0344de6bc14de66fb45dea2b82a"}
Jul 17 21:08:42.921863 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:42.921804    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:43.012339 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:43.012284    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:43.348994 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:43.348951    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/799dbb79-4b48-470c-bcec-14b0bd9c96fd-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-tlj4c\" (UID: \"799dbb79-4b48-470c-bcec-14b0bd9c96fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-tlj4c"
Jul 17 21:08:43.349169 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:43.349008    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:43.349169 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:43.349043    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:43.349169 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:43.349066    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:43.349169 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:43.349146    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-j5bcn\" (UID: \"a29d8d5f-5087-4cd7-88b3-2d79062600bf\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn"
Jul 17 21:08:43.349503 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:43.349280    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 21:08:43.349503 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:43.349346    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls podName:a29d8d5f-5087-4cd7-88b3-2d79062600bf nodeName:}" failed. No retries permitted until 2026-07-17 21:08:45.349331461 +0000 UTC m=+39.108898137 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-j5bcn" (UID: "a29d8d5f-5087-4cd7-88b3-2d79062600bf") : secret "cluster-monitoring-operator-tls" not found
Jul 17 21:08:43.350121 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:43.349747    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 21:08:43.350121 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:43.349807    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799dbb79-4b48-470c-bcec-14b0bd9c96fd-samples-operator-tls podName:799dbb79-4b48-470c-bcec-14b0bd9c96fd nodeName:}" failed. No retries permitted until 2026-07-17 21:08:45.349790278 +0000 UTC m=+39.109356953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/799dbb79-4b48-470c-bcec-14b0bd9c96fd-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-tlj4c" (UID: "799dbb79-4b48-470c-bcec-14b0bd9c96fd") : secret "samples-operator-tls" not found
Jul 17 21:08:43.350121 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:43.349882    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle podName:2cfce8d6-1da4-4003-8183-31744592793b nodeName:}" failed. No retries permitted until 2026-07-17 21:08:45.34987163 +0000 UTC m=+39.109438309 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle") pod "router-default-756f96dd8-2m7s2" (UID: "2cfce8d6-1da4-4003-8183-31744592793b") : configmap references non-existent config key: service-ca.crt
Jul 17 21:08:43.350121 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:43.349940    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 17 21:08:43.350121 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:43.349970    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs podName:2cfce8d6-1da4-4003-8183-31744592793b nodeName:}" failed. No retries permitted until 2026-07-17 21:08:45.3499604 +0000 UTC m=+39.109527074 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs") pod "router-default-756f96dd8-2m7s2" (UID: "2cfce8d6-1da4-4003-8183-31744592793b") : secret "router-metrics-certs-default" not found
Jul 17 21:08:43.350121 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:43.350031    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:08:43.350121 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:43.350041    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79fd75b69c-srz29: secret "image-registry-tls" not found
Jul 17 21:08:43.350121 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:43.350067    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls podName:43fb78e7-d06f-456a-8e39-b9fe811280aa nodeName:}" failed. No retries permitted until 2026-07-17 21:08:45.350057934 +0000 UTC m=+39.109624623 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls") pod "image-registry-79fd75b69c-srz29" (UID: "43fb78e7-d06f-456a-8e39-b9fe811280aa") : secret "image-registry-tls" not found
Jul 17 21:08:43.450294 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:43.449773    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls\") pod \"dns-default-bjqvd\" (UID: \"8859418c-d1a7-4728-989b-04addc542331\") " pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:08:43.450294 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:43.449934    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:08:43.450294 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:43.449984    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls podName:8859418c-d1a7-4728-989b-04addc542331 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:45.449971095 +0000 UTC m=+39.209537767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls") pod "dns-default-bjqvd" (UID: "8859418c-d1a7-4728-989b-04addc542331") : secret "dns-default-metrics-tls" not found
Jul 17 21:08:44.039663 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:44.039626    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:44.117146 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:44.117108    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:45.145758 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:45.145571    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:45.204102 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:45.204069    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:45.370038 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:45.369992    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-j5bcn\" (UID: \"a29d8d5f-5087-4cd7-88b3-2d79062600bf\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn"
Jul 17 21:08:45.370204 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:45.370069    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/799dbb79-4b48-470c-bcec-14b0bd9c96fd-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-tlj4c\" (UID: \"799dbb79-4b48-470c-bcec-14b0bd9c96fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-tlj4c"
Jul 17 21:08:45.370204 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:45.370091    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:45.370204 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:45.370103    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 21:08:45.370204 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:45.370118    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:45.370204 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:45.370144    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:45.370204 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:45.370176    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls podName:a29d8d5f-5087-4cd7-88b3-2d79062600bf nodeName:}" failed. No retries permitted until 2026-07-17 21:08:49.370153822 +0000 UTC m=+43.129720508 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-j5bcn" (UID: "a29d8d5f-5087-4cd7-88b3-2d79062600bf") : secret "cluster-monitoring-operator-tls" not found
Jul 17 21:08:45.370575 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:45.370221    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 21:08:45.370575 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:45.370254    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle podName:2cfce8d6-1da4-4003-8183-31744592793b nodeName:}" failed. No retries permitted until 2026-07-17 21:08:49.370238378 +0000 UTC m=+43.129805069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle") pod "router-default-756f96dd8-2m7s2" (UID: "2cfce8d6-1da4-4003-8183-31744592793b") : configmap references non-existent config key: service-ca.crt
Jul 17 21:08:45.370575 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:45.370278    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799dbb79-4b48-470c-bcec-14b0bd9c96fd-samples-operator-tls podName:799dbb79-4b48-470c-bcec-14b0bd9c96fd nodeName:}" failed. No retries permitted until 2026-07-17 21:08:49.370267866 +0000 UTC m=+43.129834539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/799dbb79-4b48-470c-bcec-14b0bd9c96fd-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-tlj4c" (UID: "799dbb79-4b48-470c-bcec-14b0bd9c96fd") : secret "samples-operator-tls" not found
Jul 17 21:08:45.370575 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:45.370285    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 17 21:08:45.370575 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:45.370353    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs podName:2cfce8d6-1da4-4003-8183-31744592793b nodeName:}" failed. No retries permitted until 2026-07-17 21:08:49.370336536 +0000 UTC m=+43.129903212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs") pod "router-default-756f96dd8-2m7s2" (UID: "2cfce8d6-1da4-4003-8183-31744592793b") : secret "router-metrics-certs-default" not found
Jul 17 21:08:45.370575 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:45.370296    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:08:45.370575 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:45.370370    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79fd75b69c-srz29: secret "image-registry-tls" not found
Jul 17 21:08:45.370575 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:45.370418    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls podName:43fb78e7-d06f-456a-8e39-b9fe811280aa nodeName:}" failed. No retries permitted until 2026-07-17 21:08:49.370392983 +0000 UTC m=+43.129959693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls") pod "image-registry-79fd75b69c-srz29" (UID: "43fb78e7-d06f-456a-8e39-b9fe811280aa") : secret "image-registry-tls" not found
Jul 17 21:08:45.471121 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:45.471050    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls\") pod \"dns-default-bjqvd\" (UID: \"8859418c-d1a7-4728-989b-04addc542331\") " pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:08:45.471249 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:45.471216    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:08:45.471293 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:45.471283    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls podName:8859418c-d1a7-4728-989b-04addc542331 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:49.471265562 +0000 UTC m=+43.230832244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls") pod "dns-default-bjqvd" (UID: "8859418c-d1a7-4728-989b-04addc542331") : secret "dns-default-metrics-tls" not found
Jul 17 21:08:45.803008 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:45.802897    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:46.952006 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:46.951798    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-pgf2z"]
Jul 17 21:08:46.972858 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:46.972831    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-pgf2z"]
Jul 17 21:08:46.973034 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:46.972955    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pgf2z"
Jul 17 21:08:46.974849 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:46.974822    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 17 21:08:46.974849 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:46.974847    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-j8bxb\""
Jul 17 21:08:46.975315 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:46.975294    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 17 21:08:47.079271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:47.079159    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:47.086691 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:47.086658    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d4b360c5-c0c1-4a87-8476-85bf4b03d169-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-pgf2z\" (UID: \"d4b360c5-c0c1-4a87-8476-85bf4b03d169\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pgf2z"
Jul 17 21:08:47.086844 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:47.086713    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pgf2z\" (UID: \"d4b360c5-c0c1-4a87-8476-85bf4b03d169\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pgf2z"
Jul 17 21:08:47.179321 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:47.179286    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:47.187256 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:47.187227    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d4b360c5-c0c1-4a87-8476-85bf4b03d169-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-pgf2z\" (UID: \"d4b360c5-c0c1-4a87-8476-85bf4b03d169\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pgf2z"
Jul 17 21:08:47.187455 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:47.187278    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pgf2z\" (UID: \"d4b360c5-c0c1-4a87-8476-85bf4b03d169\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pgf2z"
Jul 17 21:08:47.187531 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:47.187466    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 21:08:47.187584 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:47.187538    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert podName:d4b360c5-c0c1-4a87-8476-85bf4b03d169 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:47.687521004 +0000 UTC m=+41.447087676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-pgf2z" (UID: "d4b360c5-c0c1-4a87-8476-85bf4b03d169") : secret "networking-console-plugin-cert" not found
Jul 17 21:08:47.188182 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:47.188157    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d4b360c5-c0c1-4a87-8476-85bf4b03d169-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-pgf2z\" (UID: \"d4b360c5-c0c1-4a87-8476-85bf4b03d169\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pgf2z"
Jul 17 21:08:47.283897 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:47.283809    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:47.379868 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:47.379835    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:47.471562 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:47.471527    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:47.672052 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:47.672016    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:47.692894 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:47.692854    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pgf2z\" (UID: \"d4b360c5-c0c1-4a87-8476-85bf4b03d169\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pgf2z"
Jul 17 21:08:47.693074 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:47.692991    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 21:08:47.693074 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:47.693072    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert podName:d4b360c5-c0c1-4a87-8476-85bf4b03d169 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:48.693051441 +0000 UTC m=+42.452618134 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-pgf2z" (UID: "d4b360c5-c0c1-4a87-8476-85bf4b03d169") : secret "networking-console-plugin-cert" not found
Jul 17 21:08:47.735368 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:47.735332    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:48.025544 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:48.025459    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:48.235247 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:48.235213    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:48.430042 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:48.430009    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:48.488922 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:48.488889    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:48.700546 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:48.700453    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pgf2z\" (UID: \"d4b360c5-c0c1-4a87-8476-85bf4b03d169\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pgf2z"
Jul 17 21:08:48.700707 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:48.700612    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 21:08:48.700707 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:48.700696    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert podName:d4b360c5-c0c1-4a87-8476-85bf4b03d169 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:50.700673432 +0000 UTC m=+44.460240105 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-pgf2z" (UID: "d4b360c5-c0c1-4a87-8476-85bf4b03d169") : secret "networking-console-plugin-cert" not found
Jul 17 21:08:49.230519 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:49.230484    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:49.287786 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:49.287756    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:49.407679 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:49.407640    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-j5bcn\" (UID: \"a29d8d5f-5087-4cd7-88b3-2d79062600bf\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn"
Jul 17 21:08:49.407847 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:49.407721    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/799dbb79-4b48-470c-bcec-14b0bd9c96fd-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-tlj4c\" (UID: \"799dbb79-4b48-470c-bcec-14b0bd9c96fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-tlj4c"
Jul 17 21:08:49.407847 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:49.407761    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:49.407847 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:49.407792    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 21:08:49.407987 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:49.407849    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 21:08:49.407987 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:49.407860    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 17 21:08:49.407987 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:49.407856    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls podName:a29d8d5f-5087-4cd7-88b3-2d79062600bf nodeName:}" failed. No retries permitted until 2026-07-17 21:08:57.407841678 +0000 UTC m=+51.167408350 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-j5bcn" (UID: "a29d8d5f-5087-4cd7-88b3-2d79062600bf") : secret "cluster-monitoring-operator-tls" not found
Jul 17 21:08:49.407987 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:49.407909    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle podName:2cfce8d6-1da4-4003-8183-31744592793b nodeName:}" failed. No retries permitted until 2026-07-17 21:08:57.407893224 +0000 UTC m=+51.167459906 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle") pod "router-default-756f96dd8-2m7s2" (UID: "2cfce8d6-1da4-4003-8183-31744592793b") : configmap references non-existent config key: service-ca.crt
Jul 17 21:08:49.407987 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:49.407923    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799dbb79-4b48-470c-bcec-14b0bd9c96fd-samples-operator-tls podName:799dbb79-4b48-470c-bcec-14b0bd9c96fd nodeName:}" failed. No retries permitted until 2026-07-17 21:08:57.407916754 +0000 UTC m=+51.167483431 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/799dbb79-4b48-470c-bcec-14b0bd9c96fd-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-tlj4c" (UID: "799dbb79-4b48-470c-bcec-14b0bd9c96fd") : secret "samples-operator-tls" not found
Jul 17 21:08:49.407987 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:49.407934    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs podName:2cfce8d6-1da4-4003-8183-31744592793b nodeName:}" failed. No retries permitted until 2026-07-17 21:08:57.407928781 +0000 UTC m=+51.167495453 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs") pod "router-default-756f96dd8-2m7s2" (UID: "2cfce8d6-1da4-4003-8183-31744592793b") : secret "router-metrics-certs-default" not found
Jul 17 21:08:49.407987 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:49.407793    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:49.407987 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:49.407971    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:49.408237 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:49.408074    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:08:49.408237 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:49.408083    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79fd75b69c-srz29: secret "image-registry-tls" not found
Jul 17 21:08:49.408237 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:49.408114    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls podName:43fb78e7-d06f-456a-8e39-b9fe811280aa nodeName:}" failed. No retries permitted until 2026-07-17 21:08:57.408103093 +0000 UTC m=+51.167669765 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls") pod "image-registry-79fd75b69c-srz29" (UID: "43fb78e7-d06f-456a-8e39-b9fe811280aa") : secret "image-registry-tls" not found
Jul 17 21:08:49.508638 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:49.508561    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls\") pod \"dns-default-bjqvd\" (UID: \"8859418c-d1a7-4728-989b-04addc542331\") " pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:08:49.508768 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:49.508708    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:08:49.508809 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:49.508796    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls podName:8859418c-d1a7-4728-989b-04addc542331 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:57.50877992 +0000 UTC m=+51.268346602 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls") pod "dns-default-bjqvd" (UID: "8859418c-d1a7-4728-989b-04addc542331") : secret "dns-default-metrics-tls" not found
Jul 17 21:08:49.619339 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:49.619291    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:49.688756 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:49.688721    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:50.719024 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:50.718989    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pgf2z\" (UID: \"d4b360c5-c0c1-4a87-8476-85bf4b03d169\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pgf2z"
Jul 17 21:08:50.719512 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:50.719165    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 21:08:50.719512 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:50.719248    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert podName:d4b360c5-c0c1-4a87-8476-85bf4b03d169 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:54.719225683 +0000 UTC m=+48.478792368 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-pgf2z" (UID: "d4b360c5-c0c1-4a87-8476-85bf4b03d169") : secret "networking-console-plugin-cert" not found
Jul 17 21:08:51.874298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:51.874261    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-97qxw" event={"ID":"3ca96db1-3372-483e-9e19-2fb5046e81bc","Type":"ContainerStarted","Data":"cdca8c31d8b92a4348f7fe0525408a9aeaef4191131ec12ab6c6309c2041722c"}
Jul 17 21:08:51.875854 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:51.875829    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-x87k6" event={"ID":"9fd85725-3ea0-4c51-aa5f-3238c4ba6acd","Type":"ContainerStarted","Data":"bc88b8f6daa6f51d515807c526984e698ff2d715576e44aae2f2fff9b83a6a49"}
Jul 17 21:08:51.877413 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:51.877370    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tp2hx" event={"ID":"e3287b71-6904-4420-8fbb-65387da8e9f1","Type":"ContainerStarted","Data":"a3e1acdb62802da0f56156e12f2f0c8a025faac70b6bf73ec87b22c1cd0be11d"}
Jul 17 21:08:51.880371 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:51.880349    2569 generic.go:356] "Generic (PLEG): container finished" podID="9dc91e48-32c6-42ad-b3e9-7be1ec357d6f" containerID="1b28be1015376f634b8610e5075f6a4bd26192ad2d1b3ba2f3c66910691c1b52" exitCode=0
Jul 17 21:08:51.880487 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:51.880431    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxjhv" event={"ID":"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f","Type":"ContainerDied","Data":"1b28be1015376f634b8610e5075f6a4bd26192ad2d1b3ba2f3c66910691c1b52"}
Jul 17 21:08:51.881985 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:51.881961    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9szm7" event={"ID":"f596f331-2b59-4782-b6cc-ff757d3ff94c","Type":"ContainerStarted","Data":"3c7746b09b71dc6b1d97da0fece3ae8f9b716316be1947f41412f5ace0e4ef83"}
Jul 17 21:08:51.883473 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:51.883444    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gw6z9" event={"ID":"05369579-52bc-4e86-9fe2-32d608748a6d","Type":"ContainerStarted","Data":"4f7d5d82d16157937f6e6145349a132e17e08273bd51b61cf170ad61bdd30d77"}
Jul 17 21:08:51.885287 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:51.885267    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-jgv9v" event={"ID":"6878c658-1dc7-4cd2-811f-ec6a726f07ee","Type":"ContainerStarted","Data":"c700ef3ec1a713e1ecd4bf5fc5ad7c1d1cd1fa8128852c0c5e0b78798d65d30a"}
Jul 17 21:08:51.886867 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:51.886849    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/0.log"
Jul 17 21:08:51.886931 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:51.886888    2569 generic.go:356] "Generic (PLEG): container finished" podID="c0b7933f-7894-4ea9-8cbc-3704888fdeed" containerID="3edcb822d87a61434199b87b92c60def56cfc59f3b41ccc50da24e13be64928d" exitCode=255
Jul 17 21:08:51.886931 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:51.886916    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-k7c4s" event={"ID":"c0b7933f-7894-4ea9-8cbc-3704888fdeed","Type":"ContainerDied","Data":"3edcb822d87a61434199b87b92c60def56cfc59f3b41ccc50da24e13be64928d"}
Jul 17 21:08:51.887164 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:51.887147    2569 scope.go:117] "RemoveContainer" containerID="3edcb822d87a61434199b87b92c60def56cfc59f3b41ccc50da24e13be64928d"
Jul 17 21:08:51.889380 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:51.889337    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-97qxw" podStartSLOduration=4.656454186 podStartE2EDuration="10.889325294s" totalImagesPullingTime="6.232871108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:51.888459609 +0000 UTC m=+45.648026295" watchObservedRunningTime="2026-07-17 21:08:51.889325294 +0000 UTC m=+45.648891988"
Jul 17 21:08:51.923341 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:51.923079    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-x87k6" podStartSLOduration=23.270646834 podStartE2EDuration="31.923060405s" totalImagesPullingTime="8.652413571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:51.921998152 +0000 UTC m=+45.681564848" watchObservedRunningTime="2026-07-17 21:08:51.923060405 +0000 UTC m=+45.682627101"
Jul 17 21:08:51.962129 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:51.960963    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9szm7" podStartSLOduration=25.68731335 podStartE2EDuration="31.960941946s" totalImagesPullingTime="6.273628596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:51.959095441 +0000 UTC m=+45.718662136" watchObservedRunningTime="2026-07-17 21:08:51.960941946 +0000 UTC m=+45.720508622"
Jul 17 21:08:51.981315 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:51.975489    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-k7c4s"
Jul 17 21:08:51.981315 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:51.975556    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-k7c4s"
Jul 17 21:08:52.000030 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:51.999959    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tp2hx" podStartSLOduration=25.719564951 podStartE2EDuration="31.999936877s" totalImagesPullingTime="6.280371926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:51.979388294 +0000 UTC m=+45.738954987" watchObservedRunningTime="2026-07-17 21:08:51.999936877 +0000 UTC m=+45.759503572"
Jul 17 21:08:52.002059 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.001995    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:52.020099 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.018803    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-jgv9v" podStartSLOduration=23.295887522 podStartE2EDuration="32.018768665s" totalImagesPullingTime="8.722881143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:52.001158932 +0000 UTC m=+45.760725653" watchObservedRunningTime="2026-07-17 21:08:52.018768665 +0000 UTC m=+45.778335360"
Jul 17 21:08:52.098821 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.098785    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:52.162608 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.162580    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:52.355786 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.353526    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gw6z9" podStartSLOduration=23.649117001 podStartE2EDuration="32.353504125s" totalImagesPullingTime="8.704387124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:52.018531274 +0000 UTC m=+45.778097990" watchObservedRunningTime="2026-07-17 21:08:52.353504125 +0000 UTC m=+46.113070821"
Jul 17 21:08:52.355786 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.354425    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zwjfb"]
Jul 17 21:08:52.372012 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.371971    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:52.380868 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.380834    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zwjfb"]
Jul 17 21:08:52.381000 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.380962    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:08:52.382960 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.382927    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 17 21:08:52.383087 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.383058    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 17 21:08:52.383143 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.383059    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sqgsr\""
Jul 17 21:08:52.425806 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.425729    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:52.438972 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.438944    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6883c34d-3012-4a25-90d7-df75eeaf87b3-crio-socket\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:08:52.438972 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.438972    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6883c34d-3012-4a25-90d7-df75eeaf87b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:08:52.439130 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.438995    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6883c34d-3012-4a25-90d7-df75eeaf87b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:08:52.439130 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.439020    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57dcl\" (UniqueName: \"kubernetes.io/projected/6883c34d-3012-4a25-90d7-df75eeaf87b3-kube-api-access-57dcl\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:08:52.439223 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.439199    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6883c34d-3012-4a25-90d7-df75eeaf87b3-data-volume\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:08:52.540118 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.540082    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6883c34d-3012-4a25-90d7-df75eeaf87b3-data-volume\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:08:52.540283 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.540196    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6883c34d-3012-4a25-90d7-df75eeaf87b3-crio-socket\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:08:52.540283 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.540216    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6883c34d-3012-4a25-90d7-df75eeaf87b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:08:52.540283 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.540238    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6883c34d-3012-4a25-90d7-df75eeaf87b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:08:52.540283 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.540262    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57dcl\" (UniqueName: \"kubernetes.io/projected/6883c34d-3012-4a25-90d7-df75eeaf87b3-kube-api-access-57dcl\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:08:52.540528 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:52.540327    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 21:08:52.540528 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.540354    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6883c34d-3012-4a25-90d7-df75eeaf87b3-crio-socket\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:08:52.540528 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:52.540411    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 21:08:52.540528 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:52.540423    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6883c34d-3012-4a25-90d7-df75eeaf87b3-kube-rbac-proxy-cm podName:6883c34d-3012-4a25-90d7-df75eeaf87b3 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:53.040382334 +0000 UTC m=+46.799949026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6883c34d-3012-4a25-90d7-df75eeaf87b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zwjfb" (UID: "6883c34d-3012-4a25-90d7-df75eeaf87b3") : configmap "kube-rbac-proxy" not found
Jul 17 21:08:52.540528 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:52.540475    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6883c34d-3012-4a25-90d7-df75eeaf87b3-insights-runtime-extractor-tls podName:6883c34d-3012-4a25-90d7-df75eeaf87b3 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:53.040454549 +0000 UTC m=+46.800021235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6883c34d-3012-4a25-90d7-df75eeaf87b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zwjfb" (UID: "6883c34d-3012-4a25-90d7-df75eeaf87b3") : secret "insights-runtime-extractor-tls" not found
Jul 17 21:08:52.540528 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.540502    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6883c34d-3012-4a25-90d7-df75eeaf87b3-data-volume\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:08:52.550422 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.550387    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57dcl\" (UniqueName: \"kubernetes.io/projected/6883c34d-3012-4a25-90d7-df75eeaf87b3-kube-api-access-57dcl\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:08:52.725571 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.725480    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:52.893765 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.893733    2569 generic.go:356] "Generic (PLEG): container finished" podID="9dc91e48-32c6-42ad-b3e9-7be1ec357d6f" containerID="785d0e1b31f4bd4f83fb55c89ad7ceffae4241988ba80efe143471fde5892731" exitCode=0
Jul 17 21:08:52.894188 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.893823    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxjhv" event={"ID":"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f","Type":"ContainerDied","Data":"785d0e1b31f4bd4f83fb55c89ad7ceffae4241988ba80efe143471fde5892731"}
Jul 17 21:08:52.895669 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.895639    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:08:52.896058 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.896041    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/0.log"
Jul 17 21:08:52.896130 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.896075    2569 generic.go:356] "Generic (PLEG): container finished" podID="c0b7933f-7894-4ea9-8cbc-3704888fdeed" containerID="7cc5480cb86bcdaaa881dd573a2141746dcfd24f370c0dd0cdc28ec8b6ce0926" exitCode=255
Jul 17 21:08:52.896130 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.896113    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-k7c4s" event={"ID":"c0b7933f-7894-4ea9-8cbc-3704888fdeed","Type":"ContainerDied","Data":"7cc5480cb86bcdaaa881dd573a2141746dcfd24f370c0dd0cdc28ec8b6ce0926"}
Jul 17 21:08:52.896228 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.896149    2569 scope.go:117] "RemoveContainer" containerID="3edcb822d87a61434199b87b92c60def56cfc59f3b41ccc50da24e13be64928d"
Jul 17 21:08:52.896496 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.896475    2569 scope.go:117] "RemoveContainer" containerID="7cc5480cb86bcdaaa881dd573a2141746dcfd24f370c0dd0cdc28ec8b6ce0926"
Jul 17 21:08:52.896668 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:52.896645    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-k7c4s_openshift-console-operator(c0b7933f-7894-4ea9-8cbc-3704888fdeed)\"" pod="openshift-console-operator/console-operator-565b7b569d-k7c4s" podUID="c0b7933f-7894-4ea9-8cbc-3704888fdeed"
Jul 17 21:08:52.924076 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:52.924048    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:53.044494 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.044459    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6883c34d-3012-4a25-90d7-df75eeaf87b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:08:53.044681 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:53.044606    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 21:08:53.044681 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:53.044658    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 21:08:53.044681 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:53.044673    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6883c34d-3012-4a25-90d7-df75eeaf87b3-insights-runtime-extractor-tls podName:6883c34d-3012-4a25-90d7-df75eeaf87b3 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:54.04465568 +0000 UTC m=+47.804222372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6883c34d-3012-4a25-90d7-df75eeaf87b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zwjfb" (UID: "6883c34d-3012-4a25-90d7-df75eeaf87b3") : secret "insights-runtime-extractor-tls" not found
Jul 17 21:08:53.044845 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.044613    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6883c34d-3012-4a25-90d7-df75eeaf87b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:08:53.044845 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:53.044697    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6883c34d-3012-4a25-90d7-df75eeaf87b3-kube-rbac-proxy-cm podName:6883c34d-3012-4a25-90d7-df75eeaf87b3 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:54.044685941 +0000 UTC m=+47.804252613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6883c34d-3012-4a25-90d7-df75eeaf87b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zwjfb" (UID: "6883c34d-3012-4a25-90d7-df75eeaf87b3") : configmap "kube-rbac-proxy" not found
Jul 17 21:08:53.336216 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.336190    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6fvtz_98470465-ed0c-4896-8871-e8d9354e3dfa/dns-node-resolver/0.log"
Jul 17 21:08:53.390203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.390168    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:53.416598 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.416569    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-z4l8p"]
Jul 17 21:08:53.444525 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.444495    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-z4l8p"]
Jul 17 21:08:53.444667 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.444605    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-z4l8p"
Jul 17 21:08:53.446639 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.446615    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 17 21:08:53.446789 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.446685    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vzwbf\""
Jul 17 21:08:53.446837 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.446804    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 17 21:08:53.462625 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.462604    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:53.549203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.549171    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m78rm\" (UniqueName: \"kubernetes.io/projected/61b45b94-534a-419f-a4a3-424b6d8cee2a-kube-api-access-m78rm\") pod \"migrator-7cc9c7ffd7-z4l8p\" (UID: \"61b45b94-534a-419f-a4a3-424b6d8cee2a\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-z4l8p"
Jul 17 21:08:53.613937 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.613864    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:53.650300 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.650268    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78rm\" (UniqueName: \"kubernetes.io/projected/61b45b94-534a-419f-a4a3-424b6d8cee2a-kube-api-access-m78rm\") pod \"migrator-7cc9c7ffd7-z4l8p\" (UID: \"61b45b94-534a-419f-a4a3-424b6d8cee2a\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-z4l8p"
Jul 17 21:08:53.658167 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.658144    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78rm\" (UniqueName: \"kubernetes.io/projected/61b45b94-534a-419f-a4a3-424b6d8cee2a-kube-api-access-m78rm\") pod \"migrator-7cc9c7ffd7-z4l8p\" (UID: \"61b45b94-534a-419f-a4a3-424b6d8cee2a\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-z4l8p"
Jul 17 21:08:53.754018 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.753985    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-z4l8p"
Jul 17 21:08:53.897697 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.897515    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-z4l8p"]
Jul 17 21:08:53.908280 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.908252    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hxjhv" event={"ID":"9dc91e48-32c6-42ad-b3e9-7be1ec357d6f","Type":"ContainerStarted","Data":"59721aa1daeb1a82d244bc4db099db09a069787a66808d6bd8031a3eb5131893"}
Jul 17 21:08:53.909771 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.909756    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:08:53.910043 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.910028    2569 scope.go:117] "RemoveContainer" containerID="7cc5480cb86bcdaaa881dd573a2141746dcfd24f370c0dd0cdc28ec8b6ce0926"
Jul 17 21:08:53.910210 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:53.910183    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-k7c4s_openshift-console-operator(c0b7933f-7894-4ea9-8cbc-3704888fdeed)\"" pod="openshift-console-operator/console-operator-565b7b569d-k7c4s" podUID="c0b7933f-7894-4ea9-8cbc-3704888fdeed"
Jul 17 21:08:53.929277 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:53.929238    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hxjhv" podStartSLOduration=10.700099427 podStartE2EDuration="46.929226252s" totalImagesPullingTime="36.229126825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:53.926716493 +0000 UTC m=+47.686283188" watchObservedRunningTime="2026-07-17 21:08:53.929226252 +0000 UTC m=+47.688792946"
Jul 17 21:08:54.051733 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.051702    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:54.054156 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.054135    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6883c34d-3012-4a25-90d7-df75eeaf87b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:08:54.054244 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.054165    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6883c34d-3012-4a25-90d7-df75eeaf87b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:08:54.054289 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:54.054243    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 21:08:54.054326 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:54.054301    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6883c34d-3012-4a25-90d7-df75eeaf87b3-kube-rbac-proxy-cm podName:6883c34d-3012-4a25-90d7-df75eeaf87b3 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:56.05428621 +0000 UTC m=+49.813852882 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6883c34d-3012-4a25-90d7-df75eeaf87b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zwjfb" (UID: "6883c34d-3012-4a25-90d7-df75eeaf87b3") : configmap "kube-rbac-proxy" not found
Jul 17 21:08:54.054326 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:54.054303    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 21:08:54.054415 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:54.054340    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6883c34d-3012-4a25-90d7-df75eeaf87b3-insights-runtime-extractor-tls podName:6883c34d-3012-4a25-90d7-df75eeaf87b3 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:56.05432924 +0000 UTC m=+49.813895911 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6883c34d-3012-4a25-90d7-df75eeaf87b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zwjfb" (UID: "6883c34d-3012-4a25-90d7-df75eeaf87b3") : secret "insights-runtime-extractor-tls" not found
Jul 17 21:08:54.150454 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.150358    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:54.303759 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.303718    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-lzt4d"]
Jul 17 21:08:54.333489 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.333459    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-lzt4d"]
Jul 17 21:08:54.333639 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.333552    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-lzt4d"
Jul 17 21:08:54.335747 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.335715    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 17 21:08:54.335978 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.335963    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 17 21:08:54.336077 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.336049    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6z4wj\""
Jul 17 21:08:54.336077 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.336059    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 17 21:08:54.336211 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.336087    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 17 21:08:54.344309 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.344280    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:54.357950 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.357930    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/87d47a5c-5b0f-4676-9fbb-8a2fe5cadda3-signing-cabundle\") pod \"service-ca-58598695f7-lzt4d\" (UID: \"87d47a5c-5b0f-4676-9fbb-8a2fe5cadda3\") " pod="openshift-service-ca/service-ca-58598695f7-lzt4d"
Jul 17 21:08:54.358059 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.357956    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/87d47a5c-5b0f-4676-9fbb-8a2fe5cadda3-signing-key\") pod \"service-ca-58598695f7-lzt4d\" (UID: \"87d47a5c-5b0f-4676-9fbb-8a2fe5cadda3\") " pod="openshift-service-ca/service-ca-58598695f7-lzt4d"
Jul 17 21:08:54.358059 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.358030    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7csx6\" (UniqueName: \"kubernetes.io/projected/87d47a5c-5b0f-4676-9fbb-8a2fe5cadda3-kube-api-access-7csx6\") pod \"service-ca-58598695f7-lzt4d\" (UID: \"87d47a5c-5b0f-4676-9fbb-8a2fe5cadda3\") " pod="openshift-service-ca/service-ca-58598695f7-lzt4d"
Jul 17 21:08:54.398827 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.398796    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:54.459286 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.459195    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7csx6\" (UniqueName: \"kubernetes.io/projected/87d47a5c-5b0f-4676-9fbb-8a2fe5cadda3-kube-api-access-7csx6\") pod \"service-ca-58598695f7-lzt4d\" (UID: \"87d47a5c-5b0f-4676-9fbb-8a2fe5cadda3\") " pod="openshift-service-ca/service-ca-58598695f7-lzt4d"
Jul 17 21:08:54.459544 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.459517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/87d47a5c-5b0f-4676-9fbb-8a2fe5cadda3-signing-cabundle\") pod \"service-ca-58598695f7-lzt4d\" (UID: \"87d47a5c-5b0f-4676-9fbb-8a2fe5cadda3\") " pod="openshift-service-ca/service-ca-58598695f7-lzt4d"
Jul 17 21:08:54.459695 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.459556    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/87d47a5c-5b0f-4676-9fbb-8a2fe5cadda3-signing-key\") pod \"service-ca-58598695f7-lzt4d\" (UID: \"87d47a5c-5b0f-4676-9fbb-8a2fe5cadda3\") " pod="openshift-service-ca/service-ca-58598695f7-lzt4d"
Jul 17 21:08:54.460330 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.460308    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/87d47a5c-5b0f-4676-9fbb-8a2fe5cadda3-signing-cabundle\") pod \"service-ca-58598695f7-lzt4d\" (UID: \"87d47a5c-5b0f-4676-9fbb-8a2fe5cadda3\") " pod="openshift-service-ca/service-ca-58598695f7-lzt4d"
Jul 17 21:08:54.462153 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.462132    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/87d47a5c-5b0f-4676-9fbb-8a2fe5cadda3-signing-key\") pod \"service-ca-58598695f7-lzt4d\" (UID: \"87d47a5c-5b0f-4676-9fbb-8a2fe5cadda3\") " pod="openshift-service-ca/service-ca-58598695f7-lzt4d"
Jul 17 21:08:54.468603 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.468529    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:54.468785 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.468768    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7csx6\" (UniqueName: \"kubernetes.io/projected/87d47a5c-5b0f-4676-9fbb-8a2fe5cadda3-kube-api-access-7csx6\") pod \"service-ca-58598695f7-lzt4d\" (UID: \"87d47a5c-5b0f-4676-9fbb-8a2fe5cadda3\") " pod="openshift-service-ca/service-ca-58598695f7-lzt4d"
Jul 17 21:08:54.568837 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.568791    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:54.642469 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.642432    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-lzt4d"
Jul 17 21:08:54.718450 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.718367    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:54.732034 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.732015    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qn5pr_a9d9e563-636a-4912-8557-a9d71096efb8/node-ca/0.log"
Jul 17 21:08:54.762073 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.762045    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pgf2z\" (UID: \"d4b360c5-c0c1-4a87-8476-85bf4b03d169\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pgf2z"
Jul 17 21:08:54.762224 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:54.762198    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 21:08:54.762269 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:54.762260    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert podName:d4b360c5-c0c1-4a87-8476-85bf4b03d169 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:02.762244634 +0000 UTC m=+56.521811305 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-pgf2z" (UID: "d4b360c5-c0c1-4a87-8476-85bf4b03d169") : secret "networking-console-plugin-cert" not found
Jul 17 21:08:54.785766 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.785733    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-lzt4d"]
Jul 17 21:08:54.788139 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.788114    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:54.789832 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:08:54.789806    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d47a5c_5b0f_4676_9fbb_8a2fe5cadda3.slice/crio-8b7b4c0c2ab3a0a915464d7eecd80775c322612838c3dcb522d4fe0b737bd8de WatchSource:0}: Error finding container 8b7b4c0c2ab3a0a915464d7eecd80775c322612838c3dcb522d4fe0b737bd8de: Status 404 returned error can't find the container with id 8b7b4c0c2ab3a0a915464d7eecd80775c322612838c3dcb522d4fe0b737bd8de
Jul 17 21:08:54.914467 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.914435    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-z4l8p" event={"ID":"61b45b94-534a-419f-a4a3-424b6d8cee2a","Type":"ContainerStarted","Data":"b53d87a0bddb954ae974a516eaaa3315ba6030f1dde0879c7987bd77ce1daed3"}
Jul 17 21:08:54.915957 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.915920    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-lzt4d" event={"ID":"87d47a5c-5b0f-4676-9fbb-8a2fe5cadda3","Type":"ContainerStarted","Data":"8eb9e8eda15eca066571725c1e879f74a948034414753d0cb78ece08a27de76d"}
Jul 17 21:08:54.916108 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.915965    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-lzt4d" event={"ID":"87d47a5c-5b0f-4676-9fbb-8a2fe5cadda3","Type":"ContainerStarted","Data":"8b7b4c0c2ab3a0a915464d7eecd80775c322612838c3dcb522d4fe0b737bd8de"}
Jul 17 21:08:54.935241 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:54.935182    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-lzt4d" podStartSLOduration=0.935166481 podStartE2EDuration="935.166481ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:54.933838105 +0000 UTC m=+48.693404799" watchObservedRunningTime="2026-07-17 21:08:54.935166481 +0000 UTC m=+48.694733175"
Jul 17 21:08:55.132969 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:55.132940    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-97qxw_3ca96db1-3372-483e-9e19-2fb5046e81bc/serve-healthcheck-canary/0.log"
Jul 17 21:08:55.561333 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:55.561246    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:55.629184 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:55.629160    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:55.730912 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:55.730880    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:55.921318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:55.921265    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-z4l8p" event={"ID":"61b45b94-534a-419f-a4a3-424b6d8cee2a","Type":"ContainerStarted","Data":"d1ec759d9f13a6df06af3e96d37e9b5a419f15e5b0a868af50045d8b82d2b390"}
Jul 17 21:08:55.921749 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:55.921366    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-z4l8p" event={"ID":"61b45b94-534a-419f-a4a3-424b6d8cee2a","Type":"ContainerStarted","Data":"f483248a97d0cecef8c523098e19af02e25e4e2f56a114c9172c1e57016cbcab"}
Jul 17 21:08:55.956953 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:55.956903    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-z4l8p" podStartSLOduration=1.239499557 podStartE2EDuration="2.956884599s" totalImagesPullingTime="1.717385042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:55.956368906 +0000 UTC m=+49.715935603" watchObservedRunningTime="2026-07-17 21:08:55.956884599 +0000 UTC m=+49.716451294"
Jul 17 21:08:56.069167 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:56.069052    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:56.076086 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:56.076059    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6883c34d-3012-4a25-90d7-df75eeaf87b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:08:56.076239 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:56.076094    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6883c34d-3012-4a25-90d7-df75eeaf87b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:08:56.076294 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:56.076256    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 21:08:56.076358 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:56.076310    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6883c34d-3012-4a25-90d7-df75eeaf87b3-insights-runtime-extractor-tls podName:6883c34d-3012-4a25-90d7-df75eeaf87b3 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:00.076295331 +0000 UTC m=+53.835862004 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6883c34d-3012-4a25-90d7-df75eeaf87b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zwjfb" (UID: "6883c34d-3012-4a25-90d7-df75eeaf87b3") : secret "insights-runtime-extractor-tls" not found
Jul 17 21:08:56.076536 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:56.076512    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 21:08:56.076664 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:56.076575    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6883c34d-3012-4a25-90d7-df75eeaf87b3-kube-rbac-proxy-cm podName:6883c34d-3012-4a25-90d7-df75eeaf87b3 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:00.076557513 +0000 UTC m=+53.836124190 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6883c34d-3012-4a25-90d7-df75eeaf87b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zwjfb" (UID: "6883c34d-3012-4a25-90d7-df75eeaf87b3") : configmap "kube-rbac-proxy" not found
Jul 17 21:08:56.305171 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:56.305076    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:56.453926 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:56.453886    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:56.551424 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:56.551376    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:56.828729 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:56.828699    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:57.457419 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:57.457378    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:57.495476 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:57.495442    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:57.495629 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:57.495489    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:08:57.495629 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:57.495522    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:08:57.495629 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:57.495608    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:08:57.495629 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:57.495623    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79fd75b69c-srz29: secret "image-registry-tls" not found
Jul 17 21:08:57.495871 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:57.495610    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle podName:2cfce8d6-1da4-4003-8183-31744592793b nodeName:}" failed. No retries permitted until 2026-07-17 21:09:13.495590871 +0000 UTC m=+67.255157559 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle") pod "router-default-756f96dd8-2m7s2" (UID: "2cfce8d6-1da4-4003-8183-31744592793b") : configmap references non-existent config key: service-ca.crt
Jul 17 21:08:57.495871 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:57.495680    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 17 21:08:57.495871 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:57.495755    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs podName:2cfce8d6-1da4-4003-8183-31744592793b nodeName:}" failed. No retries permitted until 2026-07-17 21:09:13.495733847 +0000 UTC m=+67.255300537 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs") pod "router-default-756f96dd8-2m7s2" (UID: "2cfce8d6-1da4-4003-8183-31744592793b") : secret "router-metrics-certs-default" not found
Jul 17 21:08:57.495871 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:57.495786    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls podName:43fb78e7-d06f-456a-8e39-b9fe811280aa nodeName:}" failed. No retries permitted until 2026-07-17 21:09:13.495775785 +0000 UTC m=+67.255342457 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls") pod "image-registry-79fd75b69c-srz29" (UID: "43fb78e7-d06f-456a-8e39-b9fe811280aa") : secret "image-registry-tls" not found
Jul 17 21:08:57.495871 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:57.495860    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-j5bcn\" (UID: \"a29d8d5f-5087-4cd7-88b3-2d79062600bf\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn"
Jul 17 21:08:57.496115 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:57.495948    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/799dbb79-4b48-470c-bcec-14b0bd9c96fd-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-tlj4c\" (UID: \"799dbb79-4b48-470c-bcec-14b0bd9c96fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-tlj4c"
Jul 17 21:08:57.496115 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:57.495991    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 21:08:57.496115 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:57.496039    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls podName:a29d8d5f-5087-4cd7-88b3-2d79062600bf nodeName:}" failed. No retries permitted until 2026-07-17 21:09:13.496025784 +0000 UTC m=+67.255592471 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-j5bcn" (UID: "a29d8d5f-5087-4cd7-88b3-2d79062600bf") : secret "cluster-monitoring-operator-tls" not found
Jul 17 21:08:57.498288 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:57.498264    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/799dbb79-4b48-470c-bcec-14b0bd9c96fd-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-tlj4c\" (UID: \"799dbb79-4b48-470c-bcec-14b0bd9c96fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-tlj4c"
Jul 17 21:08:57.542705 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:57.542664    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-tlj4c"
Jul 17 21:08:57.597319 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:57.597285    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls\") pod \"dns-default-bjqvd\" (UID: \"8859418c-d1a7-4728-989b-04addc542331\") " pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:08:57.597500 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:57.597457    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:08:57.597544 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:08:57.597522    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls podName:8859418c-d1a7-4728-989b-04addc542331 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:13.597507049 +0000 UTC m=+67.357073731 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls") pod "dns-default-bjqvd" (UID: "8859418c-d1a7-4728-989b-04addc542331") : secret "dns-default-metrics-tls" not found
Jul 17 21:08:57.676212 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:57.676177    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-tlj4c"]
Jul 17 21:08:57.744231 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:57.744192    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:57.930147 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:57.930107    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-tlj4c" event={"ID":"799dbb79-4b48-470c-bcec-14b0bd9c96fd","Type":"ContainerStarted","Data":"0ed576e7db7a2b6c605d8171ea70da694b2eecd74e1ec23a1ce28225ae836ac6"}
Jul 17 21:08:58.215193 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:58.215149    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:58.347919 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:58.347821    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:58.550586 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:58.550390    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:58.771370 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:58.771332    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:59.800839 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:59.800756    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:59.863864 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:59.863832    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:08:59.939191 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:59.939153    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-tlj4c" event={"ID":"799dbb79-4b48-470c-bcec-14b0bd9c96fd","Type":"ContainerStarted","Data":"a2fa7e29893bedb005f18c81904d550bb33af9cc2eaedff33dbdd0247cbe7695"}
Jul 17 21:08:59.939352 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:59.939202    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-tlj4c" event={"ID":"799dbb79-4b48-470c-bcec-14b0bd9c96fd","Type":"ContainerStarted","Data":"87f62bba734b4c7d9e5d96d8265feb5d91f3d235259e9d546573eb689671c521"}
Jul 17 21:08:59.957461 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:08:59.957381    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-tlj4c" podStartSLOduration=38.180891979 podStartE2EDuration="39.957360641s" totalImagesPullingTime="1.776468662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:59.956986465 +0000 UTC m=+53.716553161" watchObservedRunningTime="2026-07-17 21:08:59.957360641 +0000 UTC m=+53.716927336"
Jul 17 21:09:00.122664 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:00.122626    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6883c34d-3012-4a25-90d7-df75eeaf87b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:09:00.122664 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:00.122669    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6883c34d-3012-4a25-90d7-df75eeaf87b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:09:00.122901 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:09:00.122765    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 21:09:00.122901 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:09:00.122837    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6883c34d-3012-4a25-90d7-df75eeaf87b3-insights-runtime-extractor-tls podName:6883c34d-3012-4a25-90d7-df75eeaf87b3 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:08.122808302 +0000 UTC m=+61.882374989 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6883c34d-3012-4a25-90d7-df75eeaf87b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zwjfb" (UID: "6883c34d-3012-4a25-90d7-df75eeaf87b3") : secret "insights-runtime-extractor-tls" not found
Jul 17 21:09:00.123165 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:00.123145    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6883c34d-3012-4a25-90d7-df75eeaf87b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:09:00.143329 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:00.143299    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:00.247587 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:00.247544    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:00.345128 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:00.345094    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:00.542894 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:00.542804    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:00.932529 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:00.932469    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:01.032520 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:01.032471    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:01.340975 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:01.340947    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:01.974577 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:01.974528    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-k7c4s"
Jul 17 21:09:01.974577 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:01.974596    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-k7c4s"
Jul 17 21:09:01.975089 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:01.974945    2569 scope.go:117] "RemoveContainer" containerID="7cc5480cb86bcdaaa881dd573a2141746dcfd24f370c0dd0cdc28ec8b6ce0926"
Jul 17 21:09:01.975124 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:09:01.975091    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-k7c4s_openshift-console-operator(c0b7933f-7894-4ea9-8cbc-3704888fdeed)\"" pod="openshift-console-operator/console-operator-565b7b569d-k7c4s" podUID="c0b7933f-7894-4ea9-8cbc-3704888fdeed"
Jul 17 21:09:02.182806 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:02.182771    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:02.336004 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:02.335970    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:02.736565 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:02.736483    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:02.849423 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:02.849365    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pgf2z\" (UID: \"d4b360c5-c0c1-4a87-8476-85bf4b03d169\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pgf2z"
Jul 17 21:09:02.849584 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:09:02.849511    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 21:09:02.849584 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:09:02.849576    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert podName:d4b360c5-c0c1-4a87-8476-85bf4b03d169 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:18.849560729 +0000 UTC m=+72.609127404 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-pgf2z" (UID: "d4b360c5-c0c1-4a87-8476-85bf4b03d169") : secret "networking-console-plugin-cert" not found
Jul 17 21:09:02.940690 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:02.940646    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:03.133933 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:03.133886    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:03.334397 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:03.334361    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:04.335014 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:04.334977    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:04.536132 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:04.536092    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:04.731699 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:04.731604    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:04.935587 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:04.935558    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:05.139394 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:05.139358    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:05.935101 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:05.935051    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:06.348820 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:06.348780    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:06.547328 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:06.547274    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:06.739031 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:06.738950    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:07.149765 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:07.149728    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:07.830899 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:07.830869    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ccjx"
Jul 17 21:09:08.199036 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:08.198953    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6883c34d-3012-4a25-90d7-df75eeaf87b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:09:08.201629 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:08.201583    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6883c34d-3012-4a25-90d7-df75eeaf87b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zwjfb\" (UID: \"6883c34d-3012-4a25-90d7-df75eeaf87b3\") " pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:09:08.293224 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:08.293186    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sqgsr\""
Jul 17 21:09:08.301415 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:08.301364    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zwjfb"
Jul 17 21:09:08.429292 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:08.429245    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zwjfb"]
Jul 17 21:09:08.434307 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:09:08.434273    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6883c34d_3012_4a25_90d7_df75eeaf87b3.slice/crio-af320a6fe35a278aa3b7754a4e0d74d79abd41846289bc6a6f89d165d591f462 WatchSource:0}: Error finding container af320a6fe35a278aa3b7754a4e0d74d79abd41846289bc6a6f89d165d591f462: Status 404 returned error can't find the container with id af320a6fe35a278aa3b7754a4e0d74d79abd41846289bc6a6f89d165d591f462
Jul 17 21:09:08.971318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:08.971271    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zwjfb" event={"ID":"6883c34d-3012-4a25-90d7-df75eeaf87b3","Type":"ContainerStarted","Data":"af320a6fe35a278aa3b7754a4e0d74d79abd41846289bc6a6f89d165d591f462"}
Jul 17 21:09:09.744373 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:09.744335    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:09.844912 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:09.844879    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:12.341039 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:12.340991    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4xg\" (UniqueName: \"kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg\") pod \"network-check-target-kphjs\" (UID: \"3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca\") " pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:09:12.343922 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:12.343898    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs4xg\" (UniqueName: \"kubernetes.io/projected/3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca-kube-api-access-hs4xg\") pod \"network-check-target-kphjs\" (UID: \"3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca\") " pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:09:12.442000 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:12.441959    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret\") pod \"global-pull-secret-syncer-6bxwh\" (UID: \"710a2804-cc1e-49f5-8842-d392bb9aff26\") " pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:09:12.442177 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:12.442026    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs\") pod \"network-metrics-daemon-hnhk2\" (UID: \"89bdc455-1935-456b-bc58-288a0ed415a0\") " pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:09:12.444330 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:12.444312    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 21:09:12.444386 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:12.444344    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 21:09:12.455280 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:12.455249    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89bdc455-1935-456b-bc58-288a0ed415a0-metrics-certs\") pod \"network-metrics-daemon-hnhk2\" (UID: \"89bdc455-1935-456b-bc58-288a0ed415a0\") " pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:09:12.455280 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:12.455268    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/710a2804-cc1e-49f5-8842-d392bb9aff26-original-pull-secret\") pod \"global-pull-secret-syncer-6bxwh\" (UID: \"710a2804-cc1e-49f5-8842-d392bb9aff26\") " pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:09:12.593755 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:12.593679    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8vkwx\""
Jul 17 21:09:12.598238 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:12.598206    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bxwh"
Jul 17 21:09:12.601793 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:12.601769    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:09:12.607389 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:12.607368    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-szcqz\""
Jul 17 21:09:12.616363 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:12.616343    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hnhk2"
Jul 17 21:09:13.321357 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.321156    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kphjs"]
Jul 17 21:09:13.323982 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:09:13.323940    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6c6aff_7b96_4f99_9d6a_d2ece2b9f0ca.slice/crio-c160904f85f50b1de124d4b1f67e8bfb1f85c812b1b7ae95e9ef68fe3a0909b0 WatchSource:0}: Error finding container c160904f85f50b1de124d4b1f67e8bfb1f85c812b1b7ae95e9ef68fe3a0909b0: Status 404 returned error can't find the container with id c160904f85f50b1de124d4b1f67e8bfb1f85c812b1b7ae95e9ef68fe3a0909b0
Jul 17 21:09:13.337734 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.337701    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6bxwh"]
Jul 17 21:09:13.340074 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:09:13.340044    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod710a2804_cc1e_49f5_8842_d392bb9aff26.slice/crio-24cc7f9e7dbba1ae2c7196022e93f19b47650b34417a526f545260389cda0daf WatchSource:0}: Error finding container 24cc7f9e7dbba1ae2c7196022e93f19b47650b34417a526f545260389cda0daf: Status 404 returned error can't find the container with id 24cc7f9e7dbba1ae2c7196022e93f19b47650b34417a526f545260389cda0daf
Jul 17 21:09:13.360149 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.360120    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hnhk2"]
Jul 17 21:09:13.380794 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:09:13.380765    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89bdc455_1935_456b_bc58_288a0ed415a0.slice/crio-b0fa45129038b76c1f644e7c5ec6bd77173e7f18d85f51235525f220a6fc47f1 WatchSource:0}: Error finding container b0fa45129038b76c1f644e7c5ec6bd77173e7f18d85f51235525f220a6fc47f1: Status 404 returned error can't find the container with id b0fa45129038b76c1f644e7c5ec6bd77173e7f18d85f51235525f220a6fc47f1
Jul 17 21:09:13.553740 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.553638    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:09:13.553740 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.553684    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:09:13.553982 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.553807    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:09:13.553982 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.553906    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-j5bcn\" (UID: \"a29d8d5f-5087-4cd7-88b3-2d79062600bf\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn"
Jul 17 21:09:13.554256 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.554235    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cfce8d6-1da4-4003-8183-31744592793b-service-ca-bundle\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:09:13.556094 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.556066    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cfce8d6-1da4-4003-8183-31744592793b-metrics-certs\") pod \"router-default-756f96dd8-2m7s2\" (UID: \"2cfce8d6-1da4-4003-8183-31744592793b\") " pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:09:13.556191 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.556165    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls\") pod \"image-registry-79fd75b69c-srz29\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") " pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:09:13.556314 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.556294    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a29d8d5f-5087-4cd7-88b3-2d79062600bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-j5bcn\" (UID: \"a29d8d5f-5087-4cd7-88b3-2d79062600bf\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn"
Jul 17 21:09:13.650620 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.650585    2569 scope.go:117] "RemoveContainer" containerID="7cc5480cb86bcdaaa881dd573a2141746dcfd24f370c0dd0cdc28ec8b6ce0926"
Jul 17 21:09:13.654752 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.654726    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls\") pod \"dns-default-bjqvd\" (UID: \"8859418c-d1a7-4728-989b-04addc542331\") " pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:09:13.657557 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.657535    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8859418c-d1a7-4728-989b-04addc542331-metrics-tls\") pod \"dns-default-bjqvd\" (UID: \"8859418c-d1a7-4728-989b-04addc542331\") " pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:09:13.713257 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.713229    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tdrl9\""
Jul 17 21:09:13.721227 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.721189    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn"
Jul 17 21:09:13.728753 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.728731    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cx2ql\""
Jul 17 21:09:13.737728 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.737488    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:09:13.785997 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.785134    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-b28fl\""
Jul 17 21:09:13.793901 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.793866    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:09:13.829469 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.828296    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rdmx5\""
Jul 17 21:09:13.839471 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.836041    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:09:13.933803 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.927726    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn"]
Jul 17 21:09:13.968087 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.966924    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-756f96dd8-2m7s2"]
Jul 17 21:09:13.992970 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.992939    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-756f96dd8-2m7s2" event={"ID":"2cfce8d6-1da4-4003-8183-31744592793b","Type":"ContainerStarted","Data":"c62e3168eb64e10571b9ed4da71f4f02a46f803eeaab4368b0b1e7542de3dc84"}
Jul 17 21:09:13.998717 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.998688    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:09:13.998849 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:13.998781    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-k7c4s" event={"ID":"c0b7933f-7894-4ea9-8cbc-3704888fdeed","Type":"ContainerStarted","Data":"7a611c4d988236278a9da4ab766474fea7a9a9056fddec5b2c047d14be6b957e"}
Jul 17 21:09:14.000585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:14.000542    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-k7c4s"
Jul 17 21:09:14.003896 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:14.003868    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn" event={"ID":"a29d8d5f-5087-4cd7-88b3-2d79062600bf","Type":"ContainerStarted","Data":"d307214d9635e1cc8ff1ac4a2902cb84b8cc4ce6526fe26c684846c1eaec4962"}
Jul 17 21:09:14.006989 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:14.006958    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6bxwh" event={"ID":"710a2804-cc1e-49f5-8842-d392bb9aff26","Type":"ContainerStarted","Data":"24cc7f9e7dbba1ae2c7196022e93f19b47650b34417a526f545260389cda0daf"}
Jul 17 21:09:14.009961 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:14.009936    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zwjfb" event={"ID":"6883c34d-3012-4a25-90d7-df75eeaf87b3","Type":"ContainerStarted","Data":"7a41a04662ac54d22c32a40c148ddfa63b409158cbea55a5bcb28ebea908ee61"}
Jul 17 21:09:14.018975 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:14.018894    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hnhk2" event={"ID":"89bdc455-1935-456b-bc58-288a0ed415a0","Type":"ContainerStarted","Data":"b0fa45129038b76c1f644e7c5ec6bd77173e7f18d85f51235525f220a6fc47f1"}
Jul 17 21:09:14.019768 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:09:14.019737    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43fb78e7_d06f_456a_8e39_b9fe811280aa.slice/crio-cb90590caf9ba8e9fa69e96f294600a1eeb9b8fe09a4b75442a17a32df1005b1 WatchSource:0}: Error finding container cb90590caf9ba8e9fa69e96f294600a1eeb9b8fe09a4b75442a17a32df1005b1: Status 404 returned error can't find the container with id cb90590caf9ba8e9fa69e96f294600a1eeb9b8fe09a4b75442a17a32df1005b1
Jul 17 21:09:14.020239 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:14.020201    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79fd75b69c-srz29"]
Jul 17 21:09:14.021601 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:14.021554    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-k7c4s" podStartSLOduration=47.753687299 podStartE2EDuration="54.021535705s" totalImagesPullingTime="6.267848406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:14.017904816 +0000 UTC m=+67.777471510" watchObservedRunningTime="2026-07-17 21:09:14.021535705 +0000 UTC m=+67.781102403"
Jul 17 21:09:14.023472 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:14.022735    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kphjs" event={"ID":"3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca","Type":"ContainerStarted","Data":"dd0510b9e5ad0133b509d71e42868ddd23ba154465b6e29552dfd876b7d1fd76"}
Jul 17 21:09:14.023472 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:14.022785    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kphjs" event={"ID":"3e6c6aff-7b96-4f99-9d6a-d2ece2b9f0ca","Type":"ContainerStarted","Data":"c160904f85f50b1de124d4b1f67e8bfb1f85c812b1b7ae95e9ef68fe3a0909b0"}
Jul 17 21:09:14.023472 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:14.023258    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:09:14.045820 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:14.045130    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kphjs" podStartSLOduration=68.045108105 podStartE2EDuration="1m8.045108105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:14.04317688 +0000 UTC m=+67.802743600" watchObservedRunningTime="2026-07-17 21:09:14.045108105 +0000 UTC m=+67.804674801"
Jul 17 21:09:14.050570 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:14.050522    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bjqvd"]
Jul 17 21:09:14.052621 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:09:14.052591    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8859418c_d1a7_4728_989b_04addc542331.slice/crio-feb62458fe55e41b9f50bb3669a039457b23e35098ece980e3c9a6e7f2ef0a59 WatchSource:0}: Error finding container feb62458fe55e41b9f50bb3669a039457b23e35098ece980e3c9a6e7f2ef0a59: Status 404 returned error can't find the container with id feb62458fe55e41b9f50bb3669a039457b23e35098ece980e3c9a6e7f2ef0a59
Jul 17 21:09:14.873824 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:14.873784    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-k7c4s"
Jul 17 21:09:15.032314 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:15.032247    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79fd75b69c-srz29" event={"ID":"43fb78e7-d06f-456a-8e39-b9fe811280aa","Type":"ContainerStarted","Data":"30fa3e5824b8769f380bc7c6dd9c42a2e10320f656a062bfccc5848a0069e0da"}
Jul 17 21:09:15.032314 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:15.032306    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79fd75b69c-srz29" event={"ID":"43fb78e7-d06f-456a-8e39-b9fe811280aa","Type":"ContainerStarted","Data":"cb90590caf9ba8e9fa69e96f294600a1eeb9b8fe09a4b75442a17a32df1005b1"}
Jul 17 21:09:15.033174 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:15.033077    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:09:15.035807 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:15.035745    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bjqvd" event={"ID":"8859418c-d1a7-4728-989b-04addc542331","Type":"ContainerStarted","Data":"feb62458fe55e41b9f50bb3669a039457b23e35098ece980e3c9a6e7f2ef0a59"}
Jul 17 21:09:15.041085 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:15.040676    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-756f96dd8-2m7s2" event={"ID":"2cfce8d6-1da4-4003-8183-31744592793b","Type":"ContainerStarted","Data":"450b63827ccece89e5247d03ed2f2d732539c22654e6279226e26db93e1047d5"}
Jul 17 21:09:15.067594 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:15.067531    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79fd75b69c-srz29" podStartSLOduration=55.067508777 podStartE2EDuration="55.067508777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:15.065172797 +0000 UTC m=+68.824739493" watchObservedRunningTime="2026-07-17 21:09:15.067508777 +0000 UTC m=+68.827075472"
Jul 17 21:09:15.081668 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:15.081571    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:15.092393 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:15.090872    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-756f96dd8-2m7s2" podStartSLOduration=55.090849033 podStartE2EDuration="55.090849033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:15.090166642 +0000 UTC m=+68.849733337" watchObservedRunningTime="2026-07-17 21:09:15.090849033 +0000 UTC m=+68.850415728"
Jul 17 21:09:15.182349 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:15.181931    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:15.281110 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:15.280910    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:15.381855 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:15.381815    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:15.558090 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:15.557983    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:15.738581 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:15.738503    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:09:15.741587 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:15.741365    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:09:15.931445 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:15.931278    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:15.988605 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:15.988571    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:16.048808 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:16.048769    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hnhk2" event={"ID":"89bdc455-1935-456b-bc58-288a0ed415a0","Type":"ContainerStarted","Data":"2b7e2ef1b3b65e9a9f39ee7f746701568dae536fe0501415ab6b518f7805faa9"}
Jul 17 21:09:16.048977 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:16.048838    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hnhk2" event={"ID":"89bdc455-1935-456b-bc58-288a0ed415a0","Type":"ContainerStarted","Data":"54544a6531c330309130fd4a752929e74bad29c4d2ff2445288950929cf261d1"}
Jul 17 21:09:16.048977 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:16.048870    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:09:16.050828 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:16.050804    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-756f96dd8-2m7s2"
Jul 17 21:09:16.065859 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:16.065180    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hnhk2" podStartSLOduration=67.494253328 podStartE2EDuration="1m9.065156406s" totalImagesPullingTime="1.570903078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:16.063613563 +0000 UTC m=+69.823180262" watchObservedRunningTime="2026-07-17 21:09:16.065156406 +0000 UTC m=+69.824723100"
Jul 17 21:09:16.138026 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:16.137985    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:16.366823 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:16.366788    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:16.426532 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:16.426495    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:16.565727 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:16.565645    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:16.668282 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:16.668104    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:16.767462 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:16.767414    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:17.021637 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.021544    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:17.144070 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.144038    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79fd75b69c-srz29"]
Jul 17 21:09:17.197923 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.197890    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-54vt8"]
Jul 17 21:09:17.201601 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.201573    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-54vt8"
Jul 17 21:09:17.205419 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.205374    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 17 21:09:17.205577 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.205375    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 17 21:09:17.205577 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.205476    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-t9k4h\""
Jul 17 21:09:17.215818 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.215787    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7556ff4784-ntbm5"]
Jul 17 21:09:17.219394 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.219367    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.234617 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.234584    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-54vt8"]
Jul 17 21:09:17.239996 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.239963    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7556ff4784-ntbm5"]
Jul 17 21:09:17.249562 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.249530    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:17.300227 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.299905    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf29c\" (UniqueName: \"kubernetes.io/projected/92af7adf-f9b6-4f1b-9ceb-3a915e7cb85e-kube-api-access-cf29c\") pod \"downloads-5bc67dcd46-54vt8\" (UID: \"92af7adf-f9b6-4f1b-9ceb-3a915e7cb85e\") " pod="openshift-console/downloads-5bc67dcd46-54vt8"
Jul 17 21:09:17.381843 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.381803    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:17.401175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.401145    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f1ccff32-42e3-4ce5-a863-911e4681f43f-ca-trust-extracted\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.401175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.401181    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1ccff32-42e3-4ce5-a863-911e4681f43f-trusted-ca\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.401450 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.401214    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f1ccff32-42e3-4ce5-a863-911e4681f43f-image-registry-private-configuration\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.401450 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.401270    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f1ccff32-42e3-4ce5-a863-911e4681f43f-installation-pull-secrets\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.401450 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.401330    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1ccff32-42e3-4ce5-a863-911e4681f43f-bound-sa-token\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.401450 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.401379    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f1ccff32-42e3-4ce5-a863-911e4681f43f-registry-certificates\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.401450 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.401419    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f1ccff32-42e3-4ce5-a863-911e4681f43f-registry-tls\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.401713 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.401462    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf29c\" (UniqueName: \"kubernetes.io/projected/92af7adf-f9b6-4f1b-9ceb-3a915e7cb85e-kube-api-access-cf29c\") pod \"downloads-5bc67dcd46-54vt8\" (UID: \"92af7adf-f9b6-4f1b-9ceb-3a915e7cb85e\") " pod="openshift-console/downloads-5bc67dcd46-54vt8"
Jul 17 21:09:17.401713 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.401486    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7zlk\" (UniqueName: \"kubernetes.io/projected/f1ccff32-42e3-4ce5-a863-911e4681f43f-kube-api-access-f7zlk\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.416097 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.416054    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf29c\" (UniqueName: \"kubernetes.io/projected/92af7adf-f9b6-4f1b-9ceb-3a915e7cb85e-kube-api-access-cf29c\") pod \"downloads-5bc67dcd46-54vt8\" (UID: \"92af7adf-f9b6-4f1b-9ceb-3a915e7cb85e\") " pod="openshift-console/downloads-5bc67dcd46-54vt8"
Jul 17 21:09:17.502580 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.502540    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f1ccff32-42e3-4ce5-a863-911e4681f43f-ca-trust-extracted\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.502826 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.502590    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1ccff32-42e3-4ce5-a863-911e4681f43f-trusted-ca\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.502826 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.502627    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f1ccff32-42e3-4ce5-a863-911e4681f43f-image-registry-private-configuration\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.502826 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.502652    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f1ccff32-42e3-4ce5-a863-911e4681f43f-installation-pull-secrets\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.502826 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.502679    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1ccff32-42e3-4ce5-a863-911e4681f43f-bound-sa-token\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.502826 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.502733    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f1ccff32-42e3-4ce5-a863-911e4681f43f-registry-certificates\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.502826 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.502756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f1ccff32-42e3-4ce5-a863-911e4681f43f-registry-tls\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.502826 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.502791    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7zlk\" (UniqueName: \"kubernetes.io/projected/f1ccff32-42e3-4ce5-a863-911e4681f43f-kube-api-access-f7zlk\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.503174 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.503045    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f1ccff32-42e3-4ce5-a863-911e4681f43f-ca-trust-extracted\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.504004 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.503917    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f1ccff32-42e3-4ce5-a863-911e4681f43f-registry-certificates\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.504124 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.504020    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1ccff32-42e3-4ce5-a863-911e4681f43f-trusted-ca\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.505775 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.505750    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f1ccff32-42e3-4ce5-a863-911e4681f43f-installation-pull-secrets\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.505879 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.505803    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f1ccff32-42e3-4ce5-a863-911e4681f43f-registry-tls\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.505942 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.505883    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f1ccff32-42e3-4ce5-a863-911e4681f43f-image-registry-private-configuration\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.511101 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.511067    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:17.511913 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.511893    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-54vt8"
Jul 17 21:09:17.515980 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.515949    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1ccff32-42e3-4ce5-a863-911e4681f43f-bound-sa-token\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.516384 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.516349    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7zlk\" (UniqueName: \"kubernetes.io/projected/f1ccff32-42e3-4ce5-a863-911e4681f43f-kube-api-access-f7zlk\") pod \"image-registry-7556ff4784-ntbm5\" (UID: \"f1ccff32-42e3-4ce5-a863-911e4681f43f\") " pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.530335 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.530288    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:17.608295 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.608256    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:17.702348 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:17.702308    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:18.917922 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:18.917880    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pgf2z\" (UID: \"d4b360c5-c0c1-4a87-8476-85bf4b03d169\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pgf2z"
Jul 17 21:09:18.920507 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:18.920472    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b360c5-c0c1-4a87-8476-85bf4b03d169-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pgf2z\" (UID: \"d4b360c5-c0c1-4a87-8476-85bf4b03d169\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pgf2z"
Jul 17 21:09:19.087849 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:19.087815    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-j8bxb\""
Jul 17 21:09:19.096667 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:19.096632    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pgf2z"
Jul 17 21:09:19.943685 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:19.943657    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:20.031015 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:20.030972    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:20.112689 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:20.112654    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:21.510271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:21.509290    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7556ff4784-ntbm5"]
Jul 17 21:09:21.514057 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:09:21.513982    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ccff32_42e3_4ce5_a863_911e4681f43f.slice/crio-634d0d5ec9778a3f970396f388773a123a1c5d90bfe196a151b8404f3ba9901a WatchSource:0}: Error finding container 634d0d5ec9778a3f970396f388773a123a1c5d90bfe196a151b8404f3ba9901a: Status 404 returned error can't find the container with id 634d0d5ec9778a3f970396f388773a123a1c5d90bfe196a151b8404f3ba9901a
Jul 17 21:09:21.519031 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:21.519006    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-54vt8"]
Jul 17 21:09:21.522519 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:09:21.521787    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92af7adf_f9b6_4f1b_9ceb_3a915e7cb85e.slice/crio-3ddf9a162d9f7f75b039503f8f9fb60036d3de5c6635450d3133490d433141c0 WatchSource:0}: Error finding container 3ddf9a162d9f7f75b039503f8f9fb60036d3de5c6635450d3133490d433141c0: Status 404 returned error can't find the container with id 3ddf9a162d9f7f75b039503f8f9fb60036d3de5c6635450d3133490d433141c0
Jul 17 21:09:21.535044 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:21.534979    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-pgf2z"]
Jul 17 21:09:21.663313 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:09:21.663284    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b360c5_c0c1_4a87_8476_85bf4b03d169.slice/crio-df1b9c3209262855d85005ae0194a36f5cb1a2786c350cbe0d13cff464268478 WatchSource:0}: Error finding container df1b9c3209262855d85005ae0194a36f5cb1a2786c350cbe0d13cff464268478: Status 404 returned error can't find the container with id df1b9c3209262855d85005ae0194a36f5cb1a2786c350cbe0d13cff464268478
Jul 17 21:09:21.809988 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:21.809953    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:21.908281 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:21.908247    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:21.973607 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:21.973574    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:22.077120 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:22.077077    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bjqvd" event={"ID":"8859418c-d1a7-4728-989b-04addc542331","Type":"ContainerStarted","Data":"9aab92d0ecaaa2b80d9128017952fc0acce08022e7e7feb0f8381bb53729fbf8"}
Jul 17 21:09:22.077322 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:22.077153    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bjqvd" event={"ID":"8859418c-d1a7-4728-989b-04addc542331","Type":"ContainerStarted","Data":"4931cfe8ef770af059a7f9d1a3669d2dae0d903888a1beb0bceb95289ec70042"}
Jul 17 21:09:22.077322 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:22.077185    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:09:22.078953 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:22.078915    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn" event={"ID":"a29d8d5f-5087-4cd7-88b3-2d79062600bf","Type":"ContainerStarted","Data":"1beb7de998522176c5cb4485f8d7f1d8014152a7fb6cdba3caaa3ecf1b9da5d4"}
Jul 17 21:09:22.081048 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:22.081024    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zwjfb" event={"ID":"6883c34d-3012-4a25-90d7-df75eeaf87b3","Type":"ContainerStarted","Data":"d946b35318bf4363b961b61297202b189458032c2d3dbfe988b9e742f4ae4b69"}
Jul 17 21:09:22.083336 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:22.082807    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7556ff4784-ntbm5" event={"ID":"f1ccff32-42e3-4ce5-a863-911e4681f43f","Type":"ContainerStarted","Data":"b108f7810bfad84e41bc94850fceafc1bc67a01fd71158645cd44491fc65449f"}
Jul 17 21:09:22.083336 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:22.082853    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7556ff4784-ntbm5" event={"ID":"f1ccff32-42e3-4ce5-a863-911e4681f43f","Type":"ContainerStarted","Data":"634d0d5ec9778a3f970396f388773a123a1c5d90bfe196a151b8404f3ba9901a"}
Jul 17 21:09:22.083336 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:22.082879    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:22.084447 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:22.084423    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-54vt8" event={"ID":"92af7adf-f9b6-4f1b-9ceb-3a915e7cb85e","Type":"ContainerStarted","Data":"3ddf9a162d9f7f75b039503f8f9fb60036d3de5c6635450d3133490d433141c0"}
Jul 17 21:09:22.086164 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:22.086123    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6bxwh" event={"ID":"710a2804-cc1e-49f5-8842-d392bb9aff26","Type":"ContainerStarted","Data":"493a4110e063e05a633a9029293284c6f8fa52d49888c8699ed5dccd48cc392b"}
Jul 17 21:09:22.087475 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:22.087451    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pgf2z" event={"ID":"d4b360c5-c0c1-4a87-8476-85bf4b03d169","Type":"ContainerStarted","Data":"df1b9c3209262855d85005ae0194a36f5cb1a2786c350cbe0d13cff464268478"}
Jul 17 21:09:22.131424 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:22.131336    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bjqvd" podStartSLOduration=33.878414919 podStartE2EDuration="41.131312938s" totalImagesPullingTime="7.252898019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:22.099607078 +0000 UTC m=+75.859173762" watchObservedRunningTime="2026-07-17 21:09:22.131312938 +0000 UTC m=+75.890879633"
Jul 17 21:09:22.132353 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:22.132284    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6bxwh" podStartSLOduration=67.172451498 podStartE2EDuration="1m15.13227168s" totalImagesPullingTime="7.959820182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:22.130780851 +0000 UTC m=+75.890347548" watchObservedRunningTime="2026-07-17 21:09:22.13227168 +0000 UTC m=+75.891838354"
Jul 17 21:09:22.156418 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:22.156345    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j5bcn" podStartSLOduration=54.784486297 podStartE2EDuration="1m2.156303914s" totalImagesPullingTime="7.371817617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:22.153327518 +0000 UTC m=+75.912894236" watchObservedRunningTime="2026-07-17 21:09:22.156303914 +0000 UTC m=+75.915870610"
Jul 17 21:09:22.179353 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:22.179296    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7556ff4784-ntbm5" podStartSLOduration=5.179278933 podStartE2EDuration="5.179278933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:22.177735826 +0000 UTC m=+75.937302520" watchObservedRunningTime="2026-07-17 21:09:22.179278933 +0000 UTC m=+75.938845627"
Jul 17 21:09:22.210994 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:22.210897    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:22.312931 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:22.312890    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:24.100082 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:24.100036    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pgf2z" event={"ID":"d4b360c5-c0c1-4a87-8476-85bf4b03d169","Type":"ContainerStarted","Data":"0538e649163c1e4fb5da90ec515a6c81ab3938a6c04fce58f768f68bb36b981c"}
Jul 17 21:09:24.116013 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:24.115952    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pgf2z" podStartSLOduration=36.242734481 podStartE2EDuration="38.115935448s" totalImagesPullingTime="1.873200967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:24.11427323 +0000 UTC m=+77.873839927" watchObservedRunningTime="2026-07-17 21:09:24.115935448 +0000 UTC m=+77.875502141"
Jul 17 21:09:24.242230 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:24.242186    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:24.305706 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:24.305664    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:24.717840 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:24.717798    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:24.819114 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:24.819022    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:27.116845 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:27.116804    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zwjfb" event={"ID":"6883c34d-3012-4a25-90d7-df75eeaf87b3","Type":"ContainerStarted","Data":"e8613704c02481454af692a0255844452fe265d0932acd8632e19bb86ac729da"}
Jul 17 21:09:27.134718 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:27.134666    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zwjfb" podStartSLOduration=17.3598347 podStartE2EDuration="35.134650615s" totalImagesPullingTime="17.774815915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:27.13363813 +0000 UTC m=+80.893204836" watchObservedRunningTime="2026-07-17 21:09:27.134650615 +0000 UTC m=+80.894217308"
Jul 17 21:09:27.183597 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:27.183553    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:29.163218 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:29.163178    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:29.217197 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:29.217155    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:32.095689 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:32.095653    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bjqvd"
Jul 17 21:09:33.167181 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:33.167137    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:33.225972 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:33.225934    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:33.614495 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:33.614441    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:33.666388 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:33.666322    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:34.099324 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:34.099284    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:34.199909 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:34.199871    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:34.268437 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:34.268376    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:36.114909 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:36.114871    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:36.181742 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:36.181700    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:37.017382 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:37.017332    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:37.117501 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:37.117463    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:37.203585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:37.203542    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:38.067445 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:38.067388    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:09:39.718071 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:39.718031    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:39.824236 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:39.824180    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:39.919746 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:39.919705    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:41.085463 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:41.085422    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:41.185383 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:41.185349    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:41.316844 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:41.316810    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:42.165308 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.165103    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:42.182216 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.182175    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-54vt8" event={"ID":"92af7adf-f9b6-4f1b-9ceb-3a915e7cb85e","Type":"ContainerStarted","Data":"088cdd8708ecb31b60171af631c43e68f2e996296c6abff15af29327584a2ce5"}
Jul 17 21:09:42.182503 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.182459    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-54vt8"
Jul 17 21:09:42.198451 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.198386    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-54vt8"
Jul 17 21:09:42.202179 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.202119    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-54vt8" podStartSLOduration=5.0975976 podStartE2EDuration="25.202101314s" totalImagesPullingTime="20.104503714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:42.201168408 +0000 UTC m=+95.960735199" watchObservedRunningTime="2026-07-17 21:09:42.202101314 +0000 UTC m=+95.961668009"
Jul 17 21:09:42.387162 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.387078    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:42.483237 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.483153    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-24l6g"]
Jul 17 21:09:42.484598 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.484566    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:42.489264 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.489236    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.491369 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.491344    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 17 21:09:42.491554 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.491463    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 17 21:09:42.492020 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.491856    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 17 21:09:42.492020 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.491950    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 17 21:09:42.492020 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.491968    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6lcn7\""
Jul 17 21:09:42.593255 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.593218    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:42.623198 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.623150    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6f04cb25-9d14-469d-83ba-f99620e0dead-root\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.623380 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.623211    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f04cb25-9d14-469d-83ba-f99620e0dead-sys\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.623380 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.623275    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6f04cb25-9d14-469d-83ba-f99620e0dead-node-exporter-accelerators-collector-config\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.623380 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.623311    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6f04cb25-9d14-469d-83ba-f99620e0dead-node-exporter-tls\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.623548 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.623417    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f04cb25-9d14-469d-83ba-f99620e0dead-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.623548 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.623458    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f04cb25-9d14-469d-83ba-f99620e0dead-metrics-client-ca\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.623548 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.623489    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgwmn\" (UniqueName: \"kubernetes.io/projected/6f04cb25-9d14-469d-83ba-f99620e0dead-kube-api-access-vgwmn\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.623548 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.623528    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6f04cb25-9d14-469d-83ba-f99620e0dead-node-exporter-wtmp\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.623740 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.623551    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6f04cb25-9d14-469d-83ba-f99620e0dead-node-exporter-textfile\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.724731 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.724679    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6f04cb25-9d14-469d-83ba-f99620e0dead-root\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.724922 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.724749    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f04cb25-9d14-469d-83ba-f99620e0dead-sys\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.724922 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.724793    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6f04cb25-9d14-469d-83ba-f99620e0dead-node-exporter-accelerators-collector-config\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.724922 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.724828    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6f04cb25-9d14-469d-83ba-f99620e0dead-node-exporter-tls\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.724922 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.724907    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f04cb25-9d14-469d-83ba-f99620e0dead-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.725135 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.724941    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f04cb25-9d14-469d-83ba-f99620e0dead-metrics-client-ca\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.725135 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.724966    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgwmn\" (UniqueName: \"kubernetes.io/projected/6f04cb25-9d14-469d-83ba-f99620e0dead-kube-api-access-vgwmn\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.725135 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.725005    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6f04cb25-9d14-469d-83ba-f99620e0dead-node-exporter-wtmp\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.725135 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.725039    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6f04cb25-9d14-469d-83ba-f99620e0dead-node-exporter-textfile\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.725484 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.725458    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6f04cb25-9d14-469d-83ba-f99620e0dead-node-exporter-textfile\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.725581 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.725545    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6f04cb25-9d14-469d-83ba-f99620e0dead-root\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.725635 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.725596    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f04cb25-9d14-469d-83ba-f99620e0dead-sys\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.726146 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.726120    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6f04cb25-9d14-469d-83ba-f99620e0dead-node-exporter-accelerators-collector-config\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.726257 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:09:42.726239    2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 17 21:09:42.726314 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:09:42.726305    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f04cb25-9d14-469d-83ba-f99620e0dead-node-exporter-tls podName:6f04cb25-9d14-469d-83ba-f99620e0dead nodeName:}" failed. No retries permitted until 2026-07-17 21:09:43.226284023 +0000 UTC m=+96.985850704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6f04cb25-9d14-469d-83ba-f99620e0dead-node-exporter-tls") pod "node-exporter-24l6g" (UID: "6f04cb25-9d14-469d-83ba-f99620e0dead") : secret "node-exporter-tls" not found
Jul 17 21:09:42.727125 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.727099    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6f04cb25-9d14-469d-83ba-f99620e0dead-node-exporter-wtmp\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.727544 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.727514    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f04cb25-9d14-469d-83ba-f99620e0dead-metrics-client-ca\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.730040 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.730017    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f04cb25-9d14-469d-83ba-f99620e0dead-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.738159 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.738092    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgwmn\" (UniqueName: \"kubernetes.io/projected/6f04cb25-9d14-469d-83ba-f99620e0dead-kube-api-access-vgwmn\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:42.843327 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.842872    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:42.942806 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:42.942755    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:43.044105 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.044027    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:43.084851 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.084806    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79fd75b69c-srz29" podUID="43fb78e7-d06f-456a-8e39-b9fe811280aa" containerName="registry" containerID="cri-o://30fa3e5824b8769f380bc7c6dd9c42a2e10320f656a062bfccc5848a0069e0da" gracePeriod=30
Jul 17 21:09:43.098588 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.098557    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7556ff4784-ntbm5"
Jul 17 21:09:43.230941 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.230910    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6f04cb25-9d14-469d-83ba-f99620e0dead-node-exporter-tls\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:43.234613 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.234591    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6f04cb25-9d14-469d-83ba-f99620e0dead-node-exporter-tls\") pod \"node-exporter-24l6g\" (UID: \"6f04cb25-9d14-469d-83ba-f99620e0dead\") " pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:43.273474 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.273385    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:43.374459 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.373752    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:43.402201 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.402171    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-24l6g"
Jul 17 21:09:43.443000 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:09:43.442956    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f04cb25_9d14_469d_83ba_f99620e0dead.slice/crio-f4293a0d7dc3d7462db4d780b9b7a1403f16ca572af8466c1a98fe3c1819ca49 WatchSource:0}: Error finding container f4293a0d7dc3d7462db4d780b9b7a1403f16ca572af8466c1a98fe3c1819ca49: Status 404 returned error can't find the container with id f4293a0d7dc3d7462db4d780b9b7a1403f16ca572af8466c1a98fe3c1819ca49
Jul 17 21:09:43.447204 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.447179    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:09:43.478864 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.478749    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:43.533513 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.533278    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-certificates\") pod \"43fb78e7-d06f-456a-8e39-b9fe811280aa\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") "
Jul 17 21:09:43.533513 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.533345    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/43fb78e7-d06f-456a-8e39-b9fe811280aa-image-registry-private-configuration\") pod \"43fb78e7-d06f-456a-8e39-b9fe811280aa\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") "
Jul 17 21:09:43.533751 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.533687    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-bound-sa-token\") pod \"43fb78e7-d06f-456a-8e39-b9fe811280aa\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") "
Jul 17 21:09:43.533751 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.533720    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "43fb78e7-d06f-456a-8e39-b9fe811280aa" (UID: "43fb78e7-d06f-456a-8e39-b9fe811280aa"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:09:43.533906 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.533753    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43fb78e7-d06f-456a-8e39-b9fe811280aa-trusted-ca\") pod \"43fb78e7-d06f-456a-8e39-b9fe811280aa\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") "
Jul 17 21:09:43.534102 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.534070    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43fb78e7-d06f-456a-8e39-b9fe811280aa-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "43fb78e7-d06f-456a-8e39-b9fe811280aa" (UID: "43fb78e7-d06f-456a-8e39-b9fe811280aa"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:09:43.534247 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.534189    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cm2qj\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-kube-api-access-cm2qj\") pod \"43fb78e7-d06f-456a-8e39-b9fe811280aa\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") "
Jul 17 21:09:43.534247 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.534231    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls\") pod \"43fb78e7-d06f-456a-8e39-b9fe811280aa\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") "
Jul 17 21:09:43.534363 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.534274    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/43fb78e7-d06f-456a-8e39-b9fe811280aa-installation-pull-secrets\") pod \"43fb78e7-d06f-456a-8e39-b9fe811280aa\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") "
Jul 17 21:09:43.534363 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.534317    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/43fb78e7-d06f-456a-8e39-b9fe811280aa-ca-trust-extracted\") pod \"43fb78e7-d06f-456a-8e39-b9fe811280aa\" (UID: \"43fb78e7-d06f-456a-8e39-b9fe811280aa\") "
Jul 17 21:09:43.534657 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.534609    2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-certificates\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:09:43.534657 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.534633    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/43fb78e7-d06f-456a-8e39-b9fe811280aa-trusted-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:09:43.539949 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.539895    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43fb78e7-d06f-456a-8e39-b9fe811280aa-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "43fb78e7-d06f-456a-8e39-b9fe811280aa" (UID: "43fb78e7-d06f-456a-8e39-b9fe811280aa"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:09:43.539949 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.539932    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-kube-api-access-cm2qj" (OuterVolumeSpecName: "kube-api-access-cm2qj") pod "43fb78e7-d06f-456a-8e39-b9fe811280aa" (UID: "43fb78e7-d06f-456a-8e39-b9fe811280aa"). InnerVolumeSpecName "kube-api-access-cm2qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:09:43.540426 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.540366    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43fb78e7-d06f-456a-8e39-b9fe811280aa-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "43fb78e7-d06f-456a-8e39-b9fe811280aa" (UID: "43fb78e7-d06f-456a-8e39-b9fe811280aa"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:09:43.540529 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.540334    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "43fb78e7-d06f-456a-8e39-b9fe811280aa" (UID: "43fb78e7-d06f-456a-8e39-b9fe811280aa"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:09:43.540591 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.540533    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "43fb78e7-d06f-456a-8e39-b9fe811280aa" (UID: "43fb78e7-d06f-456a-8e39-b9fe811280aa"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:09:43.548201 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.548169    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43fb78e7-d06f-456a-8e39-b9fe811280aa-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "43fb78e7-d06f-456a-8e39-b9fe811280aa" (UID: "43fb78e7-d06f-456a-8e39-b9fe811280aa"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:09:43.614695 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.614654    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:43.635508 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.635473    2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/43fb78e7-d06f-456a-8e39-b9fe811280aa-installation-pull-secrets\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:09:43.635508 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.635506    2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/43fb78e7-d06f-456a-8e39-b9fe811280aa-ca-trust-extracted\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:09:43.635745 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.635520    2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/43fb78e7-d06f-456a-8e39-b9fe811280aa-image-registry-private-configuration\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:09:43.635745 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.635535    2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-bound-sa-token\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:09:43.635745 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.635547    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm2qj\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-kube-api-access-cm2qj\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:09:43.635745 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.635559    2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/43fb78e7-d06f-456a-8e39-b9fe811280aa-registry-tls\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:09:43.690044 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.690000    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:43.874440 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.874257    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:43.934028 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:43.933983    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:44.194702 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:44.194660    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-24l6g" event={"ID":"6f04cb25-9d14-469d-83ba-f99620e0dead","Type":"ContainerStarted","Data":"f4293a0d7dc3d7462db4d780b9b7a1403f16ca572af8466c1a98fe3c1819ca49"}
Jul 17 21:09:44.196422 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:44.196360    2569 generic.go:356] "Generic (PLEG): container finished" podID="43fb78e7-d06f-456a-8e39-b9fe811280aa" containerID="30fa3e5824b8769f380bc7c6dd9c42a2e10320f656a062bfccc5848a0069e0da" exitCode=0
Jul 17 21:09:44.196564 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:44.196439    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79fd75b69c-srz29"
Jul 17 21:09:44.196564 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:44.196496    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79fd75b69c-srz29" event={"ID":"43fb78e7-d06f-456a-8e39-b9fe811280aa","Type":"ContainerDied","Data":"30fa3e5824b8769f380bc7c6dd9c42a2e10320f656a062bfccc5848a0069e0da"}
Jul 17 21:09:44.196564 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:44.196537    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79fd75b69c-srz29" event={"ID":"43fb78e7-d06f-456a-8e39-b9fe811280aa","Type":"ContainerDied","Data":"cb90590caf9ba8e9fa69e96f294600a1eeb9b8fe09a4b75442a17a32df1005b1"}
Jul 17 21:09:44.196721 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:44.196565    2569 scope.go:117] "RemoveContainer" containerID="30fa3e5824b8769f380bc7c6dd9c42a2e10320f656a062bfccc5848a0069e0da"
Jul 17 21:09:44.206975 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:44.206952    2569 scope.go:117] "RemoveContainer" containerID="30fa3e5824b8769f380bc7c6dd9c42a2e10320f656a062bfccc5848a0069e0da"
Jul 17 21:09:44.207363 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:09:44.207331    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30fa3e5824b8769f380bc7c6dd9c42a2e10320f656a062bfccc5848a0069e0da\": container with ID starting with 30fa3e5824b8769f380bc7c6dd9c42a2e10320f656a062bfccc5848a0069e0da not found: ID does not exist" containerID="30fa3e5824b8769f380bc7c6dd9c42a2e10320f656a062bfccc5848a0069e0da"
Jul 17 21:09:44.207495 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:44.207370    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30fa3e5824b8769f380bc7c6dd9c42a2e10320f656a062bfccc5848a0069e0da"} err="failed to get container status \"30fa3e5824b8769f380bc7c6dd9c42a2e10320f656a062bfccc5848a0069e0da\": rpc error: code = NotFound desc = could not find container \"30fa3e5824b8769f380bc7c6dd9c42a2e10320f656a062bfccc5848a0069e0da\": container with ID starting with 30fa3e5824b8769f380bc7c6dd9c42a2e10320f656a062bfccc5848a0069e0da not found: ID does not exist"
Jul 17 21:09:44.219657 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:44.219626    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79fd75b69c-srz29"]
Jul 17 21:09:44.223376 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:44.223350    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79fd75b69c-srz29"]
Jul 17 21:09:44.657170 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:44.657134    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43fb78e7-d06f-456a-8e39-b9fe811280aa" path="/var/lib/kubelet/pods/43fb78e7-d06f-456a-8e39-b9fe811280aa/volumes"
Jul 17 21:09:45.044429 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:45.044335    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kphjs"
Jul 17 21:09:45.387045 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:45.387012    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:45.489643 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:45.489606    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:45.570799 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:45.570770    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:45.688813 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:45.688782    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:45.788590 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:45.788519    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:46.196645 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:46.196597    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:46.209148 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:46.209115    2569 generic.go:356] "Generic (PLEG): container finished" podID="6f04cb25-9d14-469d-83ba-f99620e0dead" containerID="32d957c89ca4e26a90743e522ddaac23724ddaefc9630bf0d35100d4a1cd8e62" exitCode=0
Jul 17 21:09:46.209331 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:46.209208    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-24l6g" event={"ID":"6f04cb25-9d14-469d-83ba-f99620e0dead","Type":"ContainerDied","Data":"32d957c89ca4e26a90743e522ddaac23724ddaefc9630bf0d35100d4a1cd8e62"}
Jul 17 21:09:46.301163 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:46.301099    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:46.430966 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:46.430935    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:46.528509 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:46.528476    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:47.206699 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:47.206667    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:47.216569 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:47.216530    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-24l6g" event={"ID":"6f04cb25-9d14-469d-83ba-f99620e0dead","Type":"ContainerStarted","Data":"36845354760cd9f8c58c860c5a010fbfc2436e315539fded1659ef4893404de1"}
Jul 17 21:09:47.216748 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:47.216590    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-24l6g" event={"ID":"6f04cb25-9d14-469d-83ba-f99620e0dead","Type":"ContainerStarted","Data":"7c0b62f745bf3c5c967e118851ec86d978af41de87c0b359658a266e6f0ad6d7"}
Jul 17 21:09:47.239075 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:47.239015    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-24l6g" podStartSLOduration=3.278493718 podStartE2EDuration="5.238998634s" totalImagesPullingTime="1.855258122s" totalInitContainerRuntime="105.246794ms" isStatefulPod=true podCreationTimestamp="2026-07-17 21:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:47.237456526 +0000 UTC m=+100.997023256" watchObservedRunningTime="2026-07-17 21:09:47.238998634 +0000 UTC m=+100.998565360"
Jul 17 21:09:47.308612 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:47.308573    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:47.578623 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:47.578542    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:47.676077 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:47.676035    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:47.839807 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:47.839758    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:48.734376 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:48.734330    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:49.024755 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:49.024668    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:49.113849 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:49.113802    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:50.241909 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:50.241864    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:50.340650 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:50.340607    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:52.008073 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:52.008033    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:52.060551 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:52.060513    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:55.755627 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:55.755573    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:55.859463 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:55.859425    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:59.868674 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:59.868639    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:59.922805 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:59.922765    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:09:59.983892 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:09:59.983858    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:10:09.723930 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:10:09.723887    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:10:09.896350 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:10:09.896305    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:10:12.315520 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:10:12.315485    2569 generic.go:356] "Generic (PLEG): container finished" podID="e3287b71-6904-4420-8fbb-65387da8e9f1" containerID="a3e1acdb62802da0f56156e12f2f0c8a025faac70b6bf73ec87b22c1cd0be11d" exitCode=0
Jul 17 21:10:12.316111 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:10:12.315566    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tp2hx" event={"ID":"e3287b71-6904-4420-8fbb-65387da8e9f1","Type":"ContainerDied","Data":"a3e1acdb62802da0f56156e12f2f0c8a025faac70b6bf73ec87b22c1cd0be11d"}
Jul 17 21:10:12.316111 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:10:12.315990    2569 scope.go:117] "RemoveContainer" containerID="a3e1acdb62802da0f56156e12f2f0c8a025faac70b6bf73ec87b22c1cd0be11d"
Jul 17 21:10:13.321496 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:10:13.321453    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tp2hx" event={"ID":"e3287b71-6904-4420-8fbb-65387da8e9f1","Type":"ContainerStarted","Data":"72c0ba4f420ae569b70efc684b46cebc19c8abedbcac9d1c428b2101eeba11d0"}
Jul 17 21:10:13.322822 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:10:13.322793    2569 generic.go:356] "Generic (PLEG): container finished" podID="f596f331-2b59-4782-b6cc-ff757d3ff94c" containerID="3c7746b09b71dc6b1d97da0fece3ae8f9b716316be1947f41412f5ace0e4ef83" exitCode=0
Jul 17 21:10:13.322932 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:10:13.322859    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9szm7" event={"ID":"f596f331-2b59-4782-b6cc-ff757d3ff94c","Type":"ContainerDied","Data":"3c7746b09b71dc6b1d97da0fece3ae8f9b716316be1947f41412f5ace0e4ef83"}
Jul 17 21:10:13.323144 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:10:13.323131    2569 scope.go:117] "RemoveContainer" containerID="3c7746b09b71dc6b1d97da0fece3ae8f9b716316be1947f41412f5ace0e4ef83"
Jul 17 21:10:14.329032 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:10:14.328997    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9szm7" event={"ID":"f596f331-2b59-4782-b6cc-ff757d3ff94c","Type":"ContainerStarted","Data":"f55dbe0e8174bae8adec590e58ba60dcdc2d70b8b2404a39ae4bd3cfa287bb50"}
Jul 17 21:10:22.360646 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:10:22.360612    2569 generic.go:356] "Generic (PLEG): container finished" podID="6878c658-1dc7-4cd2-811f-ec6a726f07ee" containerID="c700ef3ec1a713e1ecd4bf5fc5ad7c1d1cd1fa8128852c0c5e0b78798d65d30a" exitCode=0
Jul 17 21:10:22.361050 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:10:22.360682    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-jgv9v" event={"ID":"6878c658-1dc7-4cd2-811f-ec6a726f07ee","Type":"ContainerDied","Data":"c700ef3ec1a713e1ecd4bf5fc5ad7c1d1cd1fa8128852c0c5e0b78798d65d30a"}
Jul 17 21:10:22.361050 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:10:22.361017    2569 scope.go:117] "RemoveContainer" containerID="c700ef3ec1a713e1ecd4bf5fc5ad7c1d1cd1fa8128852c0c5e0b78798d65d30a"
Jul 17 21:10:23.367024 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:10:23.366990    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-jgv9v" event={"ID":"6878c658-1dc7-4cd2-811f-ec6a726f07ee","Type":"ContainerStarted","Data":"c6ce4f5182c7d9a91a2d7fc65505b73a604c9d82fec133f612e86dcce76393d8"}
Jul 17 21:11:01.615982 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:11:01.615894    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:11:01.717013 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:11:01.716965    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:11:01.816848 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:11:01.816816    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:11:01.917256 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:11:01.917156    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:11:10.680551 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:11:10.680514    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:11:10.732360 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:11:10.732319    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:11:10.812958 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:11:10.812923    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:11:11.417536 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:11:11.417501    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:11:11.472084 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:11:11.472050    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:11:21.216061 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:11:21.216020    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:11:21.329809 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:11:21.329769    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:11:21.429193 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:11:21.429155    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:11:41.531319 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:11:41.531281    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:12:16.949435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:12:16.949375    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:12:17.050590 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:12:17.050554    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:12:26.893314 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:12:26.893262    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:12:26.946132 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:12:26.946074    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:12:27.304864 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:12:27.304744    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:12:27.630976 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:12:27.630941    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:12:28.969923 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:12:28.969885    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:12:29.070143 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:12:29.070109    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:12:37.522832 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:12:37.522799    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:12:37.623995 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:12:37.623943    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:12:37.675190 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:12:37.675150    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:12:39.723448 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:12:39.723413    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:12:39.823596 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:12:39.823562    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:13:06.561294 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:06.561266    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:13:06.563836 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:06.563804    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:13:06.572826 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:06.572803    2569 kubelet.go:1648] "Image garbage collection succeeded"
Jul 17 21:13:09.208357 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:09.208317    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:13:10.641098 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:10.641063    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:13:10.965999 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:10.965920    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78"]
Jul 17 21:13:10.966219 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:10.966208    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43fb78e7-d06f-456a-8e39-b9fe811280aa" containerName="registry"
Jul 17 21:13:10.966264 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:10.966221    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="43fb78e7-d06f-456a-8e39-b9fe811280aa" containerName="registry"
Jul 17 21:13:10.966297 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:10.966280    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="43fb78e7-d06f-456a-8e39-b9fe811280aa" containerName="registry"
Jul 17 21:13:10.968077 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:10.968061    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78"
Jul 17 21:13:10.970057 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:10.970033    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 17 21:13:10.970463 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:10.970446    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2plr9\""
Jul 17 21:13:10.970584 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:10.970455    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 17 21:13:10.976921 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:10.976894    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78"]
Jul 17 21:13:11.103791 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:11.103755    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2tmm\" (UniqueName: \"kubernetes.io/projected/f4cf7ba5-2362-49c7-a011-50c366b90d45-kube-api-access-r2tmm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78\" (UID: \"f4cf7ba5-2362-49c7-a011-50c366b90d45\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78"
Jul 17 21:13:11.103791 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:11.103793    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4cf7ba5-2362-49c7-a011-50c366b90d45-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78\" (UID: \"f4cf7ba5-2362-49c7-a011-50c366b90d45\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78"
Jul 17 21:13:11.104012 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:11.103830    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4cf7ba5-2362-49c7-a011-50c366b90d45-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78\" (UID: \"f4cf7ba5-2362-49c7-a011-50c366b90d45\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78"
Jul 17 21:13:11.205248 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:11.205202    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4cf7ba5-2362-49c7-a011-50c366b90d45-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78\" (UID: \"f4cf7ba5-2362-49c7-a011-50c366b90d45\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78"
Jul 17 21:13:11.205461 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:11.205285    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2tmm\" (UniqueName: \"kubernetes.io/projected/f4cf7ba5-2362-49c7-a011-50c366b90d45-kube-api-access-r2tmm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78\" (UID: \"f4cf7ba5-2362-49c7-a011-50c366b90d45\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78"
Jul 17 21:13:11.205461 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:11.205307    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4cf7ba5-2362-49c7-a011-50c366b90d45-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78\" (UID: \"f4cf7ba5-2362-49c7-a011-50c366b90d45\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78"
Jul 17 21:13:11.205654 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:11.205633    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4cf7ba5-2362-49c7-a011-50c366b90d45-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78\" (UID: \"f4cf7ba5-2362-49c7-a011-50c366b90d45\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78"
Jul 17 21:13:11.205692 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:11.205669    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4cf7ba5-2362-49c7-a011-50c366b90d45-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78\" (UID: \"f4cf7ba5-2362-49c7-a011-50c366b90d45\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78"
Jul 17 21:13:11.213976 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:11.213952    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2tmm\" (UniqueName: \"kubernetes.io/projected/f4cf7ba5-2362-49c7-a011-50c366b90d45-kube-api-access-r2tmm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78\" (UID: \"f4cf7ba5-2362-49c7-a011-50c366b90d45\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78"
Jul 17 21:13:11.277555 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:11.277465    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78"
Jul 17 21:13:11.408066 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:11.408036    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78"]
Jul 17 21:13:11.410316 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:13:11.410284    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cf7ba5_2362_49c7_a011_50c366b90d45.slice/crio-3fad1045cda5ab4d8fe8018c523aef28d1161a8bc1a8f35ae5df2b5343dba827 WatchSource:0}: Error finding container 3fad1045cda5ab4d8fe8018c523aef28d1161a8bc1a8f35ae5df2b5343dba827: Status 404 returned error can't find the container with id 3fad1045cda5ab4d8fe8018c523aef28d1161a8bc1a8f35ae5df2b5343dba827
Jul 17 21:13:11.412307 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:11.412287    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 21:13:12.000284 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:12.000239    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78" event={"ID":"f4cf7ba5-2362-49c7-a011-50c366b90d45","Type":"ContainerStarted","Data":"3fad1045cda5ab4d8fe8018c523aef28d1161a8bc1a8f35ae5df2b5343dba827"}
Jul 17 21:13:19.036593 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:19.036494    2569 generic.go:356] "Generic (PLEG): container finished" podID="f4cf7ba5-2362-49c7-a011-50c366b90d45" containerID="7459e2e7662f3481d09da0f079f09269319abb62f5ea3f63f0bcf94f6450c0c4" exitCode=0
Jul 17 21:13:19.037054 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:19.036590    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78" event={"ID":"f4cf7ba5-2362-49c7-a011-50c366b90d45","Type":"ContainerDied","Data":"7459e2e7662f3481d09da0f079f09269319abb62f5ea3f63f0bcf94f6450c0c4"}
Jul 17 21:13:49.151828 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:49.151785    2569 generic.go:356] "Generic (PLEG): container finished" podID="f4cf7ba5-2362-49c7-a011-50c366b90d45" containerID="1bf6a74a54407d760e0bbb4f9f76fff380a679475d00c0f14424669b643848bc" exitCode=0
Jul 17 21:13:49.152231 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:49.151879    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78" event={"ID":"f4cf7ba5-2362-49c7-a011-50c366b90d45","Type":"ContainerDied","Data":"1bf6a74a54407d760e0bbb4f9f76fff380a679475d00c0f14424669b643848bc"}
Jul 17 21:13:58.191449 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:58.191390    2569 generic.go:356] "Generic (PLEG): container finished" podID="f4cf7ba5-2362-49c7-a011-50c366b90d45" containerID="5f8ba10b43c62c09ba153bbaaf695b7967de62dbede130a27aa0b73b9b90a307" exitCode=0
Jul 17 21:13:58.191855 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:58.191498    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78" event={"ID":"f4cf7ba5-2362-49c7-a011-50c366b90d45","Type":"ContainerDied","Data":"5f8ba10b43c62c09ba153bbaaf695b7967de62dbede130a27aa0b73b9b90a307"}
Jul 17 21:13:59.318506 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:59.318479    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78"
Jul 17 21:13:59.409304 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:59.409267    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2tmm\" (UniqueName: \"kubernetes.io/projected/f4cf7ba5-2362-49c7-a011-50c366b90d45-kube-api-access-r2tmm\") pod \"f4cf7ba5-2362-49c7-a011-50c366b90d45\" (UID: \"f4cf7ba5-2362-49c7-a011-50c366b90d45\") "
Jul 17 21:13:59.409304 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:59.409305    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4cf7ba5-2362-49c7-a011-50c366b90d45-util\") pod \"f4cf7ba5-2362-49c7-a011-50c366b90d45\" (UID: \"f4cf7ba5-2362-49c7-a011-50c366b90d45\") "
Jul 17 21:13:59.409569 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:59.409374    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4cf7ba5-2362-49c7-a011-50c366b90d45-bundle\") pod \"f4cf7ba5-2362-49c7-a011-50c366b90d45\" (UID: \"f4cf7ba5-2362-49c7-a011-50c366b90d45\") "
Jul 17 21:13:59.410103 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:59.410076    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cf7ba5-2362-49c7-a011-50c366b90d45-bundle" (OuterVolumeSpecName: "bundle") pod "f4cf7ba5-2362-49c7-a011-50c366b90d45" (UID: "f4cf7ba5-2362-49c7-a011-50c366b90d45"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:13:59.411544 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:59.411517    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4cf7ba5-2362-49c7-a011-50c366b90d45-kube-api-access-r2tmm" (OuterVolumeSpecName: "kube-api-access-r2tmm") pod "f4cf7ba5-2362-49c7-a011-50c366b90d45" (UID: "f4cf7ba5-2362-49c7-a011-50c366b90d45"). InnerVolumeSpecName "kube-api-access-r2tmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:13:59.413455 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:59.413435    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cf7ba5-2362-49c7-a011-50c366b90d45-util" (OuterVolumeSpecName: "util") pod "f4cf7ba5-2362-49c7-a011-50c366b90d45" (UID: "f4cf7ba5-2362-49c7-a011-50c366b90d45"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:13:59.510286 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:59.510185    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4cf7ba5-2362-49c7-a011-50c366b90d45-bundle\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:13:59.510286 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:59.510232    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2tmm\" (UniqueName: \"kubernetes.io/projected/f4cf7ba5-2362-49c7-a011-50c366b90d45-kube-api-access-r2tmm\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:13:59.510286 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:13:59.510242    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4cf7ba5-2362-49c7-a011-50c366b90d45-util\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:14:00.201755 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:00.201718    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78" event={"ID":"f4cf7ba5-2362-49c7-a011-50c366b90d45","Type":"ContainerDied","Data":"3fad1045cda5ab4d8fe8018c523aef28d1161a8bc1a8f35ae5df2b5343dba827"}
Jul 17 21:14:00.201929 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:00.201766    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fad1045cda5ab4d8fe8018c523aef28d1161a8bc1a8f35ae5df2b5343dba827"
Jul 17 21:14:00.201929 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:00.201733    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xv78"
Jul 17 21:14:02.293977 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:02.293935    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:03.124221 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:03.124178    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:04.226888 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.226854    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmznc"]
Jul 17 21:14:04.227339 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.227284    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4cf7ba5-2362-49c7-a011-50c366b90d45" containerName="util"
Jul 17 21:14:04.227339 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.227307    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4cf7ba5-2362-49c7-a011-50c366b90d45" containerName="util"
Jul 17 21:14:04.227339 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.227334    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4cf7ba5-2362-49c7-a011-50c366b90d45" containerName="pull"
Jul 17 21:14:04.227528 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.227344    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4cf7ba5-2362-49c7-a011-50c366b90d45" containerName="pull"
Jul 17 21:14:04.227528 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.227359    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4cf7ba5-2362-49c7-a011-50c366b90d45" containerName="extract"
Jul 17 21:14:04.227528 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.227368    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4cf7ba5-2362-49c7-a011-50c366b90d45" containerName="extract"
Jul 17 21:14:04.227528 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.227471    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4cf7ba5-2362-49c7-a011-50c366b90d45" containerName="extract"
Jul 17 21:14:04.230465 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.230444    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmznc"
Jul 17 21:14:04.232899 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.232873    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 17 21:14:04.233034 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.232914    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:14:04.233034 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.233002    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-5252k\""
Jul 17 21:14:04.293540 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.293508    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmznc"]
Jul 17 21:14:04.344975 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.344935    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-472nv\" (UniqueName: \"kubernetes.io/projected/7fc56048-b0c0-4b52-91df-f782124eaa01-kube-api-access-472nv\") pod \"cert-manager-operator-controller-manager-55c444bf86-dmznc\" (UID: \"7fc56048-b0c0-4b52-91df-f782124eaa01\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmznc"
Jul 17 21:14:04.345171 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.344993    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fc56048-b0c0-4b52-91df-f782124eaa01-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dmznc\" (UID: \"7fc56048-b0c0-4b52-91df-f782124eaa01\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmznc"
Jul 17 21:14:04.445685 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.445645    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fc56048-b0c0-4b52-91df-f782124eaa01-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dmznc\" (UID: \"7fc56048-b0c0-4b52-91df-f782124eaa01\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmznc"
Jul 17 21:14:04.445876 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.445727    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-472nv\" (UniqueName: \"kubernetes.io/projected/7fc56048-b0c0-4b52-91df-f782124eaa01-kube-api-access-472nv\") pod \"cert-manager-operator-controller-manager-55c444bf86-dmznc\" (UID: \"7fc56048-b0c0-4b52-91df-f782124eaa01\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmznc"
Jul 17 21:14:04.446031 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.446009    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fc56048-b0c0-4b52-91df-f782124eaa01-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dmznc\" (UID: \"7fc56048-b0c0-4b52-91df-f782124eaa01\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmznc"
Jul 17 21:14:04.453950 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.453923    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-472nv\" (UniqueName: \"kubernetes.io/projected/7fc56048-b0c0-4b52-91df-f782124eaa01-kube-api-access-472nv\") pod \"cert-manager-operator-controller-manager-55c444bf86-dmznc\" (UID: \"7fc56048-b0c0-4b52-91df-f782124eaa01\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmznc"
Jul 17 21:14:04.540322 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.540232    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmznc"
Jul 17 21:14:04.677703 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:04.677592    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmznc"]
Jul 17 21:14:04.680617 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:14:04.680588    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc56048_b0c0_4b52_91df_f782124eaa01.slice/crio-2bdaa48034830a50c4ea3f853e44d8baabad3bcfea19bbc8f962dd320aafc1fd WatchSource:0}: Error finding container 2bdaa48034830a50c4ea3f853e44d8baabad3bcfea19bbc8f962dd320aafc1fd: Status 404 returned error can't find the container with id 2bdaa48034830a50c4ea3f853e44d8baabad3bcfea19bbc8f962dd320aafc1fd
Jul 17 21:14:05.223461 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:05.223395    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmznc" event={"ID":"7fc56048-b0c0-4b52-91df-f782124eaa01","Type":"ContainerStarted","Data":"2bdaa48034830a50c4ea3f853e44d8baabad3bcfea19bbc8f962dd320aafc1fd"}
Jul 17 21:14:05.903224 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:05.903188    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:06.004561 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:06.004526    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:09.154283 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:09.154241    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:09.242721 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:09.242685    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmznc" event={"ID":"7fc56048-b0c0-4b52-91df-f782124eaa01","Type":"ContainerStarted","Data":"9bd243f25ef966a26d8a5d2a25b913ec6661078eb1870fc3f456311dd13adc81"}
Jul 17 21:14:09.254131 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:09.254093    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:09.265050 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:09.265005    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmznc" podStartSLOduration=1.58318798 podStartE2EDuration="5.264992764s" totalImagesPullingTime="3.681804784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:14:09.263651984 +0000 UTC m=+363.023218702" watchObservedRunningTime="2026-07-17 21:14:09.264992764 +0000 UTC m=+363.024559458"
Jul 17 21:14:09.759936 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:09.759907    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:09.876083 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:09.876047    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:10.726356 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:10.726320    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp"]
Jul 17 21:14:10.729985 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:10.729967    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp"
Jul 17 21:14:10.732149 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:10.732116    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 17 21:14:10.732727 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:10.732703    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2plr9\""
Jul 17 21:14:10.732837 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:10.732705    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 17 21:14:10.738373 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:10.738351    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp"]
Jul 17 21:14:10.798900 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:10.798856    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06aaa788-5fd9-4217-a6f4-e031102511f0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp\" (UID: \"06aaa788-5fd9-4217-a6f4-e031102511f0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp"
Jul 17 21:14:10.799072 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:10.798954    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06aaa788-5fd9-4217-a6f4-e031102511f0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp\" (UID: \"06aaa788-5fd9-4217-a6f4-e031102511f0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp"
Jul 17 21:14:10.799072 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:10.799062    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b8sv\" (UniqueName: \"kubernetes.io/projected/06aaa788-5fd9-4217-a6f4-e031102511f0-kube-api-access-9b8sv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp\" (UID: \"06aaa788-5fd9-4217-a6f4-e031102511f0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp"
Jul 17 21:14:10.900423 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:10.900369    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06aaa788-5fd9-4217-a6f4-e031102511f0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp\" (UID: \"06aaa788-5fd9-4217-a6f4-e031102511f0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp"
Jul 17 21:14:10.900609 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:10.900438    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06aaa788-5fd9-4217-a6f4-e031102511f0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp\" (UID: \"06aaa788-5fd9-4217-a6f4-e031102511f0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp"
Jul 17 21:14:10.900609 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:10.900526    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b8sv\" (UniqueName: \"kubernetes.io/projected/06aaa788-5fd9-4217-a6f4-e031102511f0-kube-api-access-9b8sv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp\" (UID: \"06aaa788-5fd9-4217-a6f4-e031102511f0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp"
Jul 17 21:14:10.900865 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:10.900837    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06aaa788-5fd9-4217-a6f4-e031102511f0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp\" (UID: \"06aaa788-5fd9-4217-a6f4-e031102511f0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp"
Jul 17 21:14:10.900935 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:10.900846    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06aaa788-5fd9-4217-a6f4-e031102511f0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp\" (UID: \"06aaa788-5fd9-4217-a6f4-e031102511f0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp"
Jul 17 21:14:10.908478 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:10.908456    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b8sv\" (UniqueName: \"kubernetes.io/projected/06aaa788-5fd9-4217-a6f4-e031102511f0-kube-api-access-9b8sv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp\" (UID: \"06aaa788-5fd9-4217-a6f4-e031102511f0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp"
Jul 17 21:14:11.039794 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:11.039701    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp"
Jul 17 21:14:11.183310 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:14:11.183280    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06aaa788_5fd9_4217_a6f4_e031102511f0.slice/crio-86d4325a0a6d3def98abc345eace8815b4520a8297aeea6a79afabfda4ef6296 WatchSource:0}: Error finding container 86d4325a0a6d3def98abc345eace8815b4520a8297aeea6a79afabfda4ef6296: Status 404 returned error can't find the container with id 86d4325a0a6d3def98abc345eace8815b4520a8297aeea6a79afabfda4ef6296
Jul 17 21:14:11.183538 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:11.183515    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp"]
Jul 17 21:14:11.252610 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:11.252582    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp" event={"ID":"06aaa788-5fd9-4217-a6f4-e031102511f0","Type":"ContainerStarted","Data":"ea98f3c0a848634718d09b2cc04d097f60b3238e7ca172cb3bc26f3e6c421655"}
Jul 17 21:14:11.252745 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:11.252632    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp" event={"ID":"06aaa788-5fd9-4217-a6f4-e031102511f0","Type":"ContainerStarted","Data":"86d4325a0a6d3def98abc345eace8815b4520a8297aeea6a79afabfda4ef6296"}
Jul 17 21:14:11.317062 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:11.317031    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:11.414190 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:11.414153    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:12.159113 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:12.159076    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:12.258519 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:12.258421    2569 generic.go:356] "Generic (PLEG): container finished" podID="06aaa788-5fd9-4217-a6f4-e031102511f0" containerID="ea98f3c0a848634718d09b2cc04d097f60b3238e7ca172cb3bc26f3e6c421655" exitCode=0
Jul 17 21:14:12.258519 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:12.258498    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp" event={"ID":"06aaa788-5fd9-4217-a6f4-e031102511f0","Type":"ContainerDied","Data":"ea98f3c0a848634718d09b2cc04d097f60b3238e7ca172cb3bc26f3e6c421655"}
Jul 17 21:14:14.271208 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:14.271173    2569 generic.go:356] "Generic (PLEG): container finished" podID="06aaa788-5fd9-4217-a6f4-e031102511f0" containerID="37aefaea4a74132e870295c9c1b87bfa08f7f4f8416248ce9729b7ba9c693054" exitCode=0
Jul 17 21:14:14.271734 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:14.271218    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp" event={"ID":"06aaa788-5fd9-4217-a6f4-e031102511f0","Type":"ContainerDied","Data":"37aefaea4a74132e870295c9c1b87bfa08f7f4f8416248ce9729b7ba9c693054"}
Jul 17 21:14:15.277974 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:15.277933    2569 generic.go:356] "Generic (PLEG): container finished" podID="06aaa788-5fd9-4217-a6f4-e031102511f0" containerID="221c66e3bb90fb15b2d5ecd9152293dae5acb2877f54dd41140e058cb05bf7dd" exitCode=0
Jul 17 21:14:15.278347 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:15.277985    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp" event={"ID":"06aaa788-5fd9-4217-a6f4-e031102511f0","Type":"ContainerDied","Data":"221c66e3bb90fb15b2d5ecd9152293dae5acb2877f54dd41140e058cb05bf7dd"}
Jul 17 21:14:15.923755 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:15.923718    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2p64f"]
Jul 17 21:14:15.927029 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:15.927005    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2p64f"
Jul 17 21:14:15.928671 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:15.928650    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-z799x\""
Jul 17 21:14:15.928797 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:15.928676    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 17 21:14:15.929126 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:15.929109    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 17 21:14:15.935348 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:15.935327    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2p64f"]
Jul 17 21:14:16.044423 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:16.044356    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68a2a733-ee0d-4166-9410-f9d3257229d7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2p64f\" (UID: \"68a2a733-ee0d-4166-9410-f9d3257229d7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2p64f"
Jul 17 21:14:16.044600 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:16.044479    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vsdt\" (UniqueName: \"kubernetes.io/projected/68a2a733-ee0d-4166-9410-f9d3257229d7-kube-api-access-7vsdt\") pod \"cert-manager-cainjector-559cdd8c87-2p64f\" (UID: \"68a2a733-ee0d-4166-9410-f9d3257229d7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2p64f"
Jul 17 21:14:16.145676 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:16.145624    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vsdt\" (UniqueName: \"kubernetes.io/projected/68a2a733-ee0d-4166-9410-f9d3257229d7-kube-api-access-7vsdt\") pod \"cert-manager-cainjector-559cdd8c87-2p64f\" (UID: \"68a2a733-ee0d-4166-9410-f9d3257229d7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2p64f"
Jul 17 21:14:16.145887 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:16.145713    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68a2a733-ee0d-4166-9410-f9d3257229d7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2p64f\" (UID: \"68a2a733-ee0d-4166-9410-f9d3257229d7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2p64f"
Jul 17 21:14:16.153330 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:16.153297    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68a2a733-ee0d-4166-9410-f9d3257229d7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2p64f\" (UID: \"68a2a733-ee0d-4166-9410-f9d3257229d7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2p64f"
Jul 17 21:14:16.153472 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:16.153455    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vsdt\" (UniqueName: \"kubernetes.io/projected/68a2a733-ee0d-4166-9410-f9d3257229d7-kube-api-access-7vsdt\") pod \"cert-manager-cainjector-559cdd8c87-2p64f\" (UID: \"68a2a733-ee0d-4166-9410-f9d3257229d7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2p64f"
Jul 17 21:14:16.251072 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:16.250969    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2p64f"
Jul 17 21:14:16.430991 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:16.430966    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp"
Jul 17 21:14:16.549013 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:16.548920    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9b8sv\" (UniqueName: \"kubernetes.io/projected/06aaa788-5fd9-4217-a6f4-e031102511f0-kube-api-access-9b8sv\") pod \"06aaa788-5fd9-4217-a6f4-e031102511f0\" (UID: \"06aaa788-5fd9-4217-a6f4-e031102511f0\") "
Jul 17 21:14:16.549013 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:16.548964    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06aaa788-5fd9-4217-a6f4-e031102511f0-bundle\") pod \"06aaa788-5fd9-4217-a6f4-e031102511f0\" (UID: \"06aaa788-5fd9-4217-a6f4-e031102511f0\") "
Jul 17 21:14:16.549274 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:16.549020    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06aaa788-5fd9-4217-a6f4-e031102511f0-util\") pod \"06aaa788-5fd9-4217-a6f4-e031102511f0\" (UID: \"06aaa788-5fd9-4217-a6f4-e031102511f0\") "
Jul 17 21:14:16.549452 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:16.549425    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06aaa788-5fd9-4217-a6f4-e031102511f0-bundle" (OuterVolumeSpecName: "bundle") pod "06aaa788-5fd9-4217-a6f4-e031102511f0" (UID: "06aaa788-5fd9-4217-a6f4-e031102511f0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:14:16.551063 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:16.551041    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06aaa788-5fd9-4217-a6f4-e031102511f0-kube-api-access-9b8sv" (OuterVolumeSpecName: "kube-api-access-9b8sv") pod "06aaa788-5fd9-4217-a6f4-e031102511f0" (UID: "06aaa788-5fd9-4217-a6f4-e031102511f0"). InnerVolumeSpecName "kube-api-access-9b8sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:14:16.554587 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:16.554567    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06aaa788-5fd9-4217-a6f4-e031102511f0-util" (OuterVolumeSpecName: "util") pod "06aaa788-5fd9-4217-a6f4-e031102511f0" (UID: "06aaa788-5fd9-4217-a6f4-e031102511f0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:14:16.589545 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:16.589497    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2p64f"]
Jul 17 21:14:16.592326 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:14:16.592295    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68a2a733_ee0d_4166_9410_f9d3257229d7.slice/crio-55a215193f2d7967d7485d88a6b9946327ce56711a5ce42936d20a87b86615f4 WatchSource:0}: Error finding container 55a215193f2d7967d7485d88a6b9946327ce56711a5ce42936d20a87b86615f4: Status 404 returned error can't find the container with id 55a215193f2d7967d7485d88a6b9946327ce56711a5ce42936d20a87b86615f4
Jul 17 21:14:16.649562 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:16.649530    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b8sv\" (UniqueName: \"kubernetes.io/projected/06aaa788-5fd9-4217-a6f4-e031102511f0-kube-api-access-9b8sv\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:14:16.649562 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:16.649557    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06aaa788-5fd9-4217-a6f4-e031102511f0-bundle\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:14:16.649562 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:16.649568    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06aaa788-5fd9-4217-a6f4-e031102511f0-util\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:14:17.288020 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:17.287986    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp"
Jul 17 21:14:17.288247 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:17.287985    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcfbdp" event={"ID":"06aaa788-5fd9-4217-a6f4-e031102511f0","Type":"ContainerDied","Data":"86d4325a0a6d3def98abc345eace8815b4520a8297aeea6a79afabfda4ef6296"}
Jul 17 21:14:17.288247 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:17.288146    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86d4325a0a6d3def98abc345eace8815b4520a8297aeea6a79afabfda4ef6296"
Jul 17 21:14:17.289286 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:17.289261    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2p64f" event={"ID":"68a2a733-ee0d-4166-9410-f9d3257229d7","Type":"ContainerStarted","Data":"55a215193f2d7967d7485d88a6b9946327ce56711a5ce42936d20a87b86615f4"}
Jul 17 21:14:19.958156 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:19.958116    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:20.148072 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:20.148038    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:20.249001 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:20.248892    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:20.305784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:20.305750    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2p64f" event={"ID":"68a2a733-ee0d-4166-9410-f9d3257229d7","Type":"ContainerStarted","Data":"b4720aaea86bdc2c64339a294b5acdd1dc4d0e9ed6fa8b988cac40ab428e98a1"}
Jul 17 21:14:20.319900 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:20.319836    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2p64f" podStartSLOduration=2.543902299 podStartE2EDuration="5.31981569s" totalImagesPullingTime="2.775913391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:14:20.318970988 +0000 UTC m=+374.078537683" watchObservedRunningTime="2026-07-17 21:14:20.31981569 +0000 UTC m=+374.079382385"
Jul 17 21:14:21.707507 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:21.707469    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:21.803232 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:21.803200    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:29.506857 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:29.506814    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:29.608591 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:29.608528    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:31.498182 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:31.498136    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:32.391784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.391751    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl"]
Jul 17 21:14:32.392063 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.392051    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06aaa788-5fd9-4217-a6f4-e031102511f0" containerName="extract"
Jul 17 21:14:32.392106 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.392064    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="06aaa788-5fd9-4217-a6f4-e031102511f0" containerName="extract"
Jul 17 21:14:32.392106 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.392087    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06aaa788-5fd9-4217-a6f4-e031102511f0" containerName="util"
Jul 17 21:14:32.392106 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.392092    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="06aaa788-5fd9-4217-a6f4-e031102511f0" containerName="util"
Jul 17 21:14:32.392106 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.392098    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06aaa788-5fd9-4217-a6f4-e031102511f0" containerName="pull"
Jul 17 21:14:32.392106 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.392103    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="06aaa788-5fd9-4217-a6f4-e031102511f0" containerName="pull"
Jul 17 21:14:32.392252 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.392154    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="06aaa788-5fd9-4217-a6f4-e031102511f0" containerName="extract"
Jul 17 21:14:32.394439 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.394420    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl"
Jul 17 21:14:32.396367 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.396340    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 17 21:14:32.396520 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.396447    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 17 21:14:32.396520 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.396467    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2plr9\""
Jul 17 21:14:32.403246 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.403225    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl"]
Jul 17 21:14:32.483344 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.483307    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfp2c\" (UniqueName: \"kubernetes.io/projected/c52187bf-242c-4919-a0c5-7ef6bdf58c89-kube-api-access-mfp2c\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl\" (UID: \"c52187bf-242c-4919-a0c5-7ef6bdf58c89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl"
Jul 17 21:14:32.483559 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.483355    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c52187bf-242c-4919-a0c5-7ef6bdf58c89-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl\" (UID: \"c52187bf-242c-4919-a0c5-7ef6bdf58c89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl"
Jul 17 21:14:32.483559 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.483440    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c52187bf-242c-4919-a0c5-7ef6bdf58c89-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl\" (UID: \"c52187bf-242c-4919-a0c5-7ef6bdf58c89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl"
Jul 17 21:14:32.584700 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.584666    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c52187bf-242c-4919-a0c5-7ef6bdf58c89-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl\" (UID: \"c52187bf-242c-4919-a0c5-7ef6bdf58c89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl"
Jul 17 21:14:32.585167 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.584716    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c52187bf-242c-4919-a0c5-7ef6bdf58c89-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl\" (UID: \"c52187bf-242c-4919-a0c5-7ef6bdf58c89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl"
Jul 17 21:14:32.585167 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.584790    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfp2c\" (UniqueName: \"kubernetes.io/projected/c52187bf-242c-4919-a0c5-7ef6bdf58c89-kube-api-access-mfp2c\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl\" (UID: \"c52187bf-242c-4919-a0c5-7ef6bdf58c89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl"
Jul 17 21:14:32.585167 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.585032    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c52187bf-242c-4919-a0c5-7ef6bdf58c89-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl\" (UID: \"c52187bf-242c-4919-a0c5-7ef6bdf58c89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl"
Jul 17 21:14:32.585167 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.585078    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c52187bf-242c-4919-a0c5-7ef6bdf58c89-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl\" (UID: \"c52187bf-242c-4919-a0c5-7ef6bdf58c89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl"
Jul 17 21:14:32.592053 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.592032    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfp2c\" (UniqueName: \"kubernetes.io/projected/c52187bf-242c-4919-a0c5-7ef6bdf58c89-kube-api-access-mfp2c\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl\" (UID: \"c52187bf-242c-4919-a0c5-7ef6bdf58c89\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl"
Jul 17 21:14:32.704205 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.704118    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl"
Jul 17 21:14:32.835099 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:32.835071    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl"]
Jul 17 21:14:32.837851 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:14:32.837820    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52187bf_242c_4919_a0c5_7ef6bdf58c89.slice/crio-799c8eda22607c01536d0881fbfbc2e0de56359287c66b9aa32837e3ff0c1bd0 WatchSource:0}: Error finding container 799c8eda22607c01536d0881fbfbc2e0de56359287c66b9aa32837e3ff0c1bd0: Status 404 returned error can't find the container with id 799c8eda22607c01536d0881fbfbc2e0de56359287c66b9aa32837e3ff0c1bd0
Jul 17 21:14:33.365285 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:33.365251    2569 generic.go:356] "Generic (PLEG): container finished" podID="c52187bf-242c-4919-a0c5-7ef6bdf58c89" containerID="b371dcced6740b811a1ad65686b4edb6d1d1033d69c8de5fb6d66136ad452ffb" exitCode=0
Jul 17 21:14:33.365478 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:33.365336    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl" event={"ID":"c52187bf-242c-4919-a0c5-7ef6bdf58c89","Type":"ContainerDied","Data":"b371dcced6740b811a1ad65686b4edb6d1d1033d69c8de5fb6d66136ad452ffb"}
Jul 17 21:14:33.365478 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:33.365381    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl" event={"ID":"c52187bf-242c-4919-a0c5-7ef6bdf58c89","Type":"ContainerStarted","Data":"799c8eda22607c01536d0881fbfbc2e0de56359287c66b9aa32837e3ff0c1bd0"}
Jul 17 21:14:34.370938 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:34.370908    2569 generic.go:356] "Generic (PLEG): container finished" podID="c52187bf-242c-4919-a0c5-7ef6bdf58c89" containerID="3a37830aab763a146d4215c28733c347226e422d0ba27a41c9aac5044a8cdea9" exitCode=0
Jul 17 21:14:34.371316 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:34.370989    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl" event={"ID":"c52187bf-242c-4919-a0c5-7ef6bdf58c89","Type":"ContainerDied","Data":"3a37830aab763a146d4215c28733c347226e422d0ba27a41c9aac5044a8cdea9"}
Jul 17 21:14:35.377331 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:35.377293    2569 generic.go:356] "Generic (PLEG): container finished" podID="c52187bf-242c-4919-a0c5-7ef6bdf58c89" containerID="43bf8752bdc15f2ce5807ff05c2fc62dd7b00e716dbb509562eb85c284edbb17" exitCode=0
Jul 17 21:14:35.377747 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:35.377375    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl" event={"ID":"c52187bf-242c-4919-a0c5-7ef6bdf58c89","Type":"ContainerDied","Data":"43bf8752bdc15f2ce5807ff05c2fc62dd7b00e716dbb509562eb85c284edbb17"}
Jul 17 21:14:36.514475 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:36.514449    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl"
Jul 17 21:14:36.617247 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:36.617205    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c52187bf-242c-4919-a0c5-7ef6bdf58c89-util\") pod \"c52187bf-242c-4919-a0c5-7ef6bdf58c89\" (UID: \"c52187bf-242c-4919-a0c5-7ef6bdf58c89\") "
Jul 17 21:14:36.617476 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:36.617255    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfp2c\" (UniqueName: \"kubernetes.io/projected/c52187bf-242c-4919-a0c5-7ef6bdf58c89-kube-api-access-mfp2c\") pod \"c52187bf-242c-4919-a0c5-7ef6bdf58c89\" (UID: \"c52187bf-242c-4919-a0c5-7ef6bdf58c89\") "
Jul 17 21:14:36.617476 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:36.617297    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c52187bf-242c-4919-a0c5-7ef6bdf58c89-bundle\") pod \"c52187bf-242c-4919-a0c5-7ef6bdf58c89\" (UID: \"c52187bf-242c-4919-a0c5-7ef6bdf58c89\") "
Jul 17 21:14:36.618075 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:36.618046    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52187bf-242c-4919-a0c5-7ef6bdf58c89-bundle" (OuterVolumeSpecName: "bundle") pod "c52187bf-242c-4919-a0c5-7ef6bdf58c89" (UID: "c52187bf-242c-4919-a0c5-7ef6bdf58c89"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:14:36.619417 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:36.619377    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52187bf-242c-4919-a0c5-7ef6bdf58c89-kube-api-access-mfp2c" (OuterVolumeSpecName: "kube-api-access-mfp2c") pod "c52187bf-242c-4919-a0c5-7ef6bdf58c89" (UID: "c52187bf-242c-4919-a0c5-7ef6bdf58c89"). InnerVolumeSpecName "kube-api-access-mfp2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:14:36.624925 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:36.624899    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52187bf-242c-4919-a0c5-7ef6bdf58c89-util" (OuterVolumeSpecName: "util") pod "c52187bf-242c-4919-a0c5-7ef6bdf58c89" (UID: "c52187bf-242c-4919-a0c5-7ef6bdf58c89"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:14:36.717888 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:36.717860    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c52187bf-242c-4919-a0c5-7ef6bdf58c89-util\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:14:36.717888 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:36.717889    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfp2c\" (UniqueName: \"kubernetes.io/projected/c52187bf-242c-4919-a0c5-7ef6bdf58c89-kube-api-access-mfp2c\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:14:36.718130 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:36.717904    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c52187bf-242c-4919-a0c5-7ef6bdf58c89-bundle\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:14:37.393188 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:37.393155    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl"
Jul 17 21:14:37.393188 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:37.393172    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznjsl" event={"ID":"c52187bf-242c-4919-a0c5-7ef6bdf58c89","Type":"ContainerDied","Data":"799c8eda22607c01536d0881fbfbc2e0de56359287c66b9aa32837e3ff0c1bd0"}
Jul 17 21:14:37.393467 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:37.393219    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="799c8eda22607c01536d0881fbfbc2e0de56359287c66b9aa32837e3ff0c1bd0"
Jul 17 21:14:40.437032 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:40.436993    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:43.507521 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:43.507482    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:43.601538 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:43.601506    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:44.115275 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.115229    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:44.408594 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.408495    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:44.507288 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.507256    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:44.551293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.551256    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"]
Jul 17 21:14:44.551673 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.551577    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52187bf-242c-4919-a0c5-7ef6bdf58c89" containerName="util"
Jul 17 21:14:44.551673 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.551589    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52187bf-242c-4919-a0c5-7ef6bdf58c89" containerName="util"
Jul 17 21:14:44.551673 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.551597    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52187bf-242c-4919-a0c5-7ef6bdf58c89" containerName="extract"
Jul 17 21:14:44.551673 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.551604    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52187bf-242c-4919-a0c5-7ef6bdf58c89" containerName="extract"
Jul 17 21:14:44.551673 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.551611    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52187bf-242c-4919-a0c5-7ef6bdf58c89" containerName="pull"
Jul 17 21:14:44.551673 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.551616    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52187bf-242c-4919-a0c5-7ef6bdf58c89" containerName="pull"
Jul 17 21:14:44.551871 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.551682    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c52187bf-242c-4919-a0c5-7ef6bdf58c89" containerName="extract"
Jul 17 21:14:44.557241 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.557221    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"
Jul 17 21:14:44.561831 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.561811    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 17 21:14:44.562132 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.562116    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 17 21:14:44.562312 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.562282    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 17 21:14:44.563079 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.563059    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rvgtv\""
Jul 17 21:14:44.563344 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.563328    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 17 21:14:44.572632 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.572607    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"]
Jul 17 21:14:44.683807 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.683720    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/abe9dd49-3a15-4f58-b19f-e871479de840-manifests\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-jqdz6\" (UID: \"abe9dd49-3a15-4f58-b19f-e871479de840\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"
Jul 17 21:14:44.683807 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.683760    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdjkk\" (UniqueName: \"kubernetes.io/projected/abe9dd49-3a15-4f58-b19f-e871479de840-kube-api-access-hdjkk\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-jqdz6\" (UID: \"abe9dd49-3a15-4f58-b19f-e871479de840\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"
Jul 17 21:14:44.683807 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.683796    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abe9dd49-3a15-4f58-b19f-e871479de840-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-jqdz6\" (UID: \"abe9dd49-3a15-4f58-b19f-e871479de840\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"
Jul 17 21:14:44.684017 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.683832    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/abe9dd49-3a15-4f58-b19f-e871479de840-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-jqdz6\" (UID: \"abe9dd49-3a15-4f58-b19f-e871479de840\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"
Jul 17 21:14:44.684017 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.683856    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abe9dd49-3a15-4f58-b19f-e871479de840-tmp\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-jqdz6\" (UID: \"abe9dd49-3a15-4f58-b19f-e871479de840\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"
Jul 17 21:14:44.686204 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.686166    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:44.785119 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.785081    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/abe9dd49-3a15-4f58-b19f-e871479de840-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-jqdz6\" (UID: \"abe9dd49-3a15-4f58-b19f-e871479de840\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"
Jul 17 21:14:44.785282 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.785126    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abe9dd49-3a15-4f58-b19f-e871479de840-tmp\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-jqdz6\" (UID: \"abe9dd49-3a15-4f58-b19f-e871479de840\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"
Jul 17 21:14:44.785282 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.785203    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/abe9dd49-3a15-4f58-b19f-e871479de840-manifests\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-jqdz6\" (UID: \"abe9dd49-3a15-4f58-b19f-e871479de840\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"
Jul 17 21:14:44.785282 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.785244    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdjkk\" (UniqueName: \"kubernetes.io/projected/abe9dd49-3a15-4f58-b19f-e871479de840-kube-api-access-hdjkk\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-jqdz6\" (UID: \"abe9dd49-3a15-4f58-b19f-e871479de840\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"
Jul 17 21:14:44.785282 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.785275    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abe9dd49-3a15-4f58-b19f-e871479de840-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-jqdz6\" (UID: \"abe9dd49-3a15-4f58-b19f-e871479de840\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"
Jul 17 21:14:44.785722 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.785691    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/abe9dd49-3a15-4f58-b19f-e871479de840-manifests\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-jqdz6\" (UID: \"abe9dd49-3a15-4f58-b19f-e871479de840\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"
Jul 17 21:14:44.787532 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.787495    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:44.787893 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.787876    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abe9dd49-3a15-4f58-b19f-e871479de840-tmp\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-jqdz6\" (UID: \"abe9dd49-3a15-4f58-b19f-e871479de840\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"
Jul 17 21:14:44.788110 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.788092    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/abe9dd49-3a15-4f58-b19f-e871479de840-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-jqdz6\" (UID: \"abe9dd49-3a15-4f58-b19f-e871479de840\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"
Jul 17 21:14:44.788181 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.788162    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abe9dd49-3a15-4f58-b19f-e871479de840-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-jqdz6\" (UID: \"abe9dd49-3a15-4f58-b19f-e871479de840\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"
Jul 17 21:14:44.797462 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.797434    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdjkk\" (UniqueName: \"kubernetes.io/projected/abe9dd49-3a15-4f58-b19f-e871479de840-kube-api-access-hdjkk\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-jqdz6\" (UID: \"abe9dd49-3a15-4f58-b19f-e871479de840\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"
Jul 17 21:14:44.867968 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.867931    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"
Jul 17 21:14:44.886644 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.886613    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:44.989187 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:44.989156    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:45.004968 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:45.004932    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"]
Jul 17 21:14:45.009969 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:14:45.009933    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe9dd49_3a15_4f58_b19f_e871479de840.slice/crio-3f4821a37023a3f38ccecc94f0437485f183d22f6f61c30489661fa8b883c6fa WatchSource:0}: Error finding container 3f4821a37023a3f38ccecc94f0437485f183d22f6f61c30489661fa8b883c6fa: Status 404 returned error can't find the container with id 3f4821a37023a3f38ccecc94f0437485f183d22f6f61c30489661fa8b883c6fa
Jul 17 21:14:45.106649 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:45.106608    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:45.428761 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:45.428726    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6" event={"ID":"abe9dd49-3a15-4f58-b19f-e871479de840","Type":"ContainerStarted","Data":"3f4821a37023a3f38ccecc94f0437485f183d22f6f61c30489661fa8b883c6fa"}
Jul 17 21:14:48.444658 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:48.444620    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6" event={"ID":"abe9dd49-3a15-4f58-b19f-e871479de840","Type":"ContainerStarted","Data":"d2afb872a0aa61b7a57d0499a0406a92c62db4931a0c153a2db416c4e84dcf9d"}
Jul 17 21:14:49.450179 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:49.450142    2569 generic.go:356] "Generic (PLEG): container finished" podID="abe9dd49-3a15-4f58-b19f-e871479de840" containerID="d2afb872a0aa61b7a57d0499a0406a92c62db4931a0c153a2db416c4e84dcf9d" exitCode=0
Jul 17 21:14:49.450568 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:49.450232    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6" event={"ID":"abe9dd49-3a15-4f58-b19f-e871479de840","Type":"ContainerDied","Data":"d2afb872a0aa61b7a57d0499a0406a92c62db4931a0c153a2db416c4e84dcf9d"}
Jul 17 21:14:50.456887 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:50.456850    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6" event={"ID":"abe9dd49-3a15-4f58-b19f-e871479de840","Type":"ContainerStarted","Data":"e3baeee45f686044ee9060ac4e9b8ac07a4b7ef022c940132461b85a27a7cd5c"}
Jul 17 21:14:50.457272 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:50.456930    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"
Jul 17 21:14:50.476816 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:50.476771    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6" podStartSLOduration=1.9245508839999999 podStartE2EDuration="6.476755009s" totalImagesPullingTime="3.521921708s" totalInitContainerRuntime="1.030282417s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:14:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:14:50.47575533 +0000 UTC m=+404.235322023" watchObservedRunningTime="2026-07-17 21:14:50.476755009 +0000 UTC m=+404.236321703"
Jul 17 21:14:56.329781 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:56.329737    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:14:56.434964 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:14:56.434924    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:01.464186 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:01.464153    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-jqdz6"
Jul 17 21:15:03.622975 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:03.622931    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:03.744862 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:03.744827    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:04.049044 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.048925    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:04.186611 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.186572    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7cf476465c-wbzp5"]
Jul 17 21:15:04.190146 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.190112    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cf476465c-wbzp5"
Jul 17 21:15:04.193341 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.193313    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ptj6w\""
Jul 17 21:15:04.193772 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.193750    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 17 21:15:04.193927 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.193766    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 17 21:15:04.203172 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.203141    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cf476465c-wbzp5"]
Jul 17 21:15:04.249070 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.249037    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efda4d41-9563-4e36-b789-859c4786e297-tmp\") pod \"kube-auth-proxy-7cf476465c-wbzp5\" (UID: \"efda4d41-9563-4e36-b789-859c4786e297\") " pod="openshift-ingress/kube-auth-proxy-7cf476465c-wbzp5"
Jul 17 21:15:04.249336 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.249088    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efda4d41-9563-4e36-b789-859c4786e297-tls-certs\") pod \"kube-auth-proxy-7cf476465c-wbzp5\" (UID: \"efda4d41-9563-4e36-b789-859c4786e297\") " pod="openshift-ingress/kube-auth-proxy-7cf476465c-wbzp5"
Jul 17 21:15:04.249336 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.249114    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hrcg\" (UniqueName: \"kubernetes.io/projected/efda4d41-9563-4e36-b789-859c4786e297-kube-api-access-8hrcg\") pod \"kube-auth-proxy-7cf476465c-wbzp5\" (UID: \"efda4d41-9563-4e36-b789-859c4786e297\") " pod="openshift-ingress/kube-auth-proxy-7cf476465c-wbzp5"
Jul 17 21:15:04.350326 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.350274    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efda4d41-9563-4e36-b789-859c4786e297-tmp\") pod \"kube-auth-proxy-7cf476465c-wbzp5\" (UID: \"efda4d41-9563-4e36-b789-859c4786e297\") " pod="openshift-ingress/kube-auth-proxy-7cf476465c-wbzp5"
Jul 17 21:15:04.350542 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.350342    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efda4d41-9563-4e36-b789-859c4786e297-tls-certs\") pod \"kube-auth-proxy-7cf476465c-wbzp5\" (UID: \"efda4d41-9563-4e36-b789-859c4786e297\") " pod="openshift-ingress/kube-auth-proxy-7cf476465c-wbzp5"
Jul 17 21:15:04.350542 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:15:04.350470    2569 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 17 21:15:04.350542 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.350506    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hrcg\" (UniqueName: \"kubernetes.io/projected/efda4d41-9563-4e36-b789-859c4786e297-kube-api-access-8hrcg\") pod \"kube-auth-proxy-7cf476465c-wbzp5\" (UID: \"efda4d41-9563-4e36-b789-859c4786e297\") " pod="openshift-ingress/kube-auth-proxy-7cf476465c-wbzp5"
Jul 17 21:15:04.350542 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:15:04.350535    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efda4d41-9563-4e36-b789-859c4786e297-tls-certs podName:efda4d41-9563-4e36-b789-859c4786e297 nodeName:}" failed. No retries permitted until 2026-07-17 21:15:04.850518859 +0000 UTC m=+418.610085530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/efda4d41-9563-4e36-b789-859c4786e297-tls-certs") pod "kube-auth-proxy-7cf476465c-wbzp5" (UID: "efda4d41-9563-4e36-b789-859c4786e297") : secret "kube-auth-proxy-tls" not found
Jul 17 21:15:04.352893 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.352860    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efda4d41-9563-4e36-b789-859c4786e297-tmp\") pod \"kube-auth-proxy-7cf476465c-wbzp5\" (UID: \"efda4d41-9563-4e36-b789-859c4786e297\") " pod="openshift-ingress/kube-auth-proxy-7cf476465c-wbzp5"
Jul 17 21:15:04.371136 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.371102    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hrcg\" (UniqueName: \"kubernetes.io/projected/efda4d41-9563-4e36-b789-859c4786e297-kube-api-access-8hrcg\") pod \"kube-auth-proxy-7cf476465c-wbzp5\" (UID: \"efda4d41-9563-4e36-b789-859c4786e297\") " pod="openshift-ingress/kube-auth-proxy-7cf476465c-wbzp5"
Jul 17 21:15:04.387559 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.387520    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r8sf5"]
Jul 17 21:15:04.391320 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.391294    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-r8sf5"
Jul 17 21:15:04.393609 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.393585    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 17 21:15:04.393737 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.393695    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-gttsm\""
Jul 17 21:15:04.405642 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.405614    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r8sf5"]
Jul 17 21:15:04.452060 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.452024    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78n72\" (UniqueName: \"kubernetes.io/projected/aaf6b929-c2d5-48fb-ab6b-535386eb9635-kube-api-access-78n72\") pod \"odh-model-controller-7998c67dc4-r8sf5\" (UID: \"aaf6b929-c2d5-48fb-ab6b-535386eb9635\") " pod="opendatahub/odh-model-controller-7998c67dc4-r8sf5"
Jul 17 21:15:04.452235 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.452074    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaf6b929-c2d5-48fb-ab6b-535386eb9635-cert\") pod \"odh-model-controller-7998c67dc4-r8sf5\" (UID: \"aaf6b929-c2d5-48fb-ab6b-535386eb9635\") " pod="opendatahub/odh-model-controller-7998c67dc4-r8sf5"
Jul 17 21:15:04.474582 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.474538    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:04.553359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.553319    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78n72\" (UniqueName: \"kubernetes.io/projected/aaf6b929-c2d5-48fb-ab6b-535386eb9635-kube-api-access-78n72\") pod \"odh-model-controller-7998c67dc4-r8sf5\" (UID: \"aaf6b929-c2d5-48fb-ab6b-535386eb9635\") " pod="opendatahub/odh-model-controller-7998c67dc4-r8sf5"
Jul 17 21:15:04.553359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.553361    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaf6b929-c2d5-48fb-ab6b-535386eb9635-cert\") pod \"odh-model-controller-7998c67dc4-r8sf5\" (UID: \"aaf6b929-c2d5-48fb-ab6b-535386eb9635\") " pod="opendatahub/odh-model-controller-7998c67dc4-r8sf5"
Jul 17 21:15:04.553664 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:15:04.553502    2569 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 17 21:15:04.553664 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:15:04.553589    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaf6b929-c2d5-48fb-ab6b-535386eb9635-cert podName:aaf6b929-c2d5-48fb-ab6b-535386eb9635 nodeName:}" failed. No retries permitted until 2026-07-17 21:15:05.053569588 +0000 UTC m=+418.813136263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/aaf6b929-c2d5-48fb-ab6b-535386eb9635-cert") pod "odh-model-controller-7998c67dc4-r8sf5" (UID: "aaf6b929-c2d5-48fb-ab6b-535386eb9635") : secret "odh-model-controller-webhook-cert" not found
Jul 17 21:15:04.564262 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.564221    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78n72\" (UniqueName: \"kubernetes.io/projected/aaf6b929-c2d5-48fb-ab6b-535386eb9635-kube-api-access-78n72\") pod \"odh-model-controller-7998c67dc4-r8sf5\" (UID: \"aaf6b929-c2d5-48fb-ab6b-535386eb9635\") " pod="opendatahub/odh-model-controller-7998c67dc4-r8sf5"
Jul 17 21:15:04.576336 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.576268    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:04.856085 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.856043    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efda4d41-9563-4e36-b789-859c4786e297-tls-certs\") pod \"kube-auth-proxy-7cf476465c-wbzp5\" (UID: \"efda4d41-9563-4e36-b789-859c4786e297\") " pod="openshift-ingress/kube-auth-proxy-7cf476465c-wbzp5"
Jul 17 21:15:04.858877 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:04.858848    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efda4d41-9563-4e36-b789-859c4786e297-tls-certs\") pod \"kube-auth-proxy-7cf476465c-wbzp5\" (UID: \"efda4d41-9563-4e36-b789-859c4786e297\") " pod="openshift-ingress/kube-auth-proxy-7cf476465c-wbzp5"
Jul 17 21:15:05.058222 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.058172    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaf6b929-c2d5-48fb-ab6b-535386eb9635-cert\") pod \"odh-model-controller-7998c67dc4-r8sf5\" (UID: \"aaf6b929-c2d5-48fb-ab6b-535386eb9635\") " pod="opendatahub/odh-model-controller-7998c67dc4-r8sf5"
Jul 17 21:15:05.061147 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.061110    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aaf6b929-c2d5-48fb-ab6b-535386eb9635-cert\") pod \"odh-model-controller-7998c67dc4-r8sf5\" (UID: \"aaf6b929-c2d5-48fb-ab6b-535386eb9635\") " pod="opendatahub/odh-model-controller-7998c67dc4-r8sf5"
Jul 17 21:15:05.069837 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.069790    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:05.101084 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.101051    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cf476465c-wbzp5"
Jul 17 21:15:05.181558 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.181521    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:05.183542 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.183512    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lms8h"]
Jul 17 21:15:05.187174 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.187150    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-lms8h"
Jul 17 21:15:05.190133 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.189973    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 17 21:15:05.190133 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.190003    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-2sfgn\""
Jul 17 21:15:05.231954 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.231915    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lms8h"]
Jul 17 21:15:05.259877 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.259840    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2106c72f-6b22-4c1b-8328-9fb93b711349-tls-certs\") pod \"model-serving-api-54d89bd5fc-lms8h\" (UID: \"2106c72f-6b22-4c1b-8328-9fb93b711349\") " pod="opendatahub/model-serving-api-54d89bd5fc-lms8h"
Jul 17 21:15:05.259877 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.259882    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk5bh\" (UniqueName: \"kubernetes.io/projected/2106c72f-6b22-4c1b-8328-9fb93b711349-kube-api-access-tk5bh\") pod \"model-serving-api-54d89bd5fc-lms8h\" (UID: \"2106c72f-6b22-4c1b-8328-9fb93b711349\") " pod="opendatahub/model-serving-api-54d89bd5fc-lms8h"
Jul 17 21:15:05.300871 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.300823    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cf476465c-wbzp5"]
Jul 17 21:15:05.303201 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.303169    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-r8sf5"
Jul 17 21:15:05.312536 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:15:05.312501    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefda4d41_9563_4e36_b789_859c4786e297.slice/crio-d83072374389635640e3546e5d8a10d3e0e13ab8b9e8a5001b1a9ff397fc54e5 WatchSource:0}: Error finding container d83072374389635640e3546e5d8a10d3e0e13ab8b9e8a5001b1a9ff397fc54e5: Status 404 returned error can't find the container with id d83072374389635640e3546e5d8a10d3e0e13ab8b9e8a5001b1a9ff397fc54e5
Jul 17 21:15:05.361476 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.361440    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2106c72f-6b22-4c1b-8328-9fb93b711349-tls-certs\") pod \"model-serving-api-54d89bd5fc-lms8h\" (UID: \"2106c72f-6b22-4c1b-8328-9fb93b711349\") " pod="opendatahub/model-serving-api-54d89bd5fc-lms8h"
Jul 17 21:15:05.361639 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.361503    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk5bh\" (UniqueName: \"kubernetes.io/projected/2106c72f-6b22-4c1b-8328-9fb93b711349-kube-api-access-tk5bh\") pod \"model-serving-api-54d89bd5fc-lms8h\" (UID: \"2106c72f-6b22-4c1b-8328-9fb93b711349\") " pod="opendatahub/model-serving-api-54d89bd5fc-lms8h"
Jul 17 21:15:05.366041 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.365703    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2106c72f-6b22-4c1b-8328-9fb93b711349-tls-certs\") pod \"model-serving-api-54d89bd5fc-lms8h\" (UID: \"2106c72f-6b22-4c1b-8328-9fb93b711349\") " pod="opendatahub/model-serving-api-54d89bd5fc-lms8h"
Jul 17 21:15:05.375760 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.375726    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk5bh\" (UniqueName: \"kubernetes.io/projected/2106c72f-6b22-4c1b-8328-9fb93b711349-kube-api-access-tk5bh\") pod \"model-serving-api-54d89bd5fc-lms8h\" (UID: \"2106c72f-6b22-4c1b-8328-9fb93b711349\") " pod="opendatahub/model-serving-api-54d89bd5fc-lms8h"
Jul 17 21:15:05.393894 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.393657    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:05.446243 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.446216    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r8sf5"]
Jul 17 21:15:05.448440 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:15:05.448411    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf6b929_c2d5_48fb_ab6b_535386eb9635.slice/crio-ab8a8c783ba3ce91cff882493da89ca95585f2594e0a43801dcfb30a0d96af15 WatchSource:0}: Error finding container ab8a8c783ba3ce91cff882493da89ca95585f2594e0a43801dcfb30a0d96af15: Status 404 returned error can't find the container with id ab8a8c783ba3ce91cff882493da89ca95585f2594e0a43801dcfb30a0d96af15
Jul 17 21:15:05.501456 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.501415    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-lms8h"
Jul 17 21:15:05.526900 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.526858    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cf476465c-wbzp5" event={"ID":"efda4d41-9563-4e36-b789-859c4786e297","Type":"ContainerStarted","Data":"d83072374389635640e3546e5d8a10d3e0e13ab8b9e8a5001b1a9ff397fc54e5"}
Jul 17 21:15:05.528049 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.528024    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-r8sf5" event={"ID":"aaf6b929-c2d5-48fb-ab6b-535386eb9635","Type":"ContainerStarted","Data":"ab8a8c783ba3ce91cff882493da89ca95585f2594e0a43801dcfb30a0d96af15"}
Jul 17 21:15:05.650475 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.650371    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"]
Jul 17 21:15:05.653794 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.653776    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.656892 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.656868    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-lms8h"]
Jul 17 21:15:05.658349 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.658325    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-d9p27\""
Jul 17 21:15:05.658349 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.658344    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 17 21:15:05.658648 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.658629    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 17 21:15:05.658777 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.658657    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 17 21:15:05.658777 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.658662    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 17 21:15:05.659460 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:15:05.659381    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2106c72f_6b22_4c1b_8328_9fb93b711349.slice/crio-7e615838f3ba6c9cc6d6364f0b50e1990534b93ecc001ee6e8b4d977e897c4da WatchSource:0}: Error finding container 7e615838f3ba6c9cc6d6364f0b50e1990534b93ecc001ee6e8b4d977e897c4da: Status 404 returned error can't find the container with id 7e615838f3ba6c9cc6d6364f0b50e1990534b93ecc001ee6e8b4d977e897c4da
Jul 17 21:15:05.671779 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.671745    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"]
Jul 17 21:15:05.764495 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.764440    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-token\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.764712 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.764551    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.764712 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.764629    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.764712 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.764678    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e55def1-38d7-4e6e-a775-48df1f8890aa-local-certs\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.764879 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.764716    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.764879 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.764739    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jbrx\" (UniqueName: \"kubernetes.io/projected/4e55def1-38d7-4e6e-a775-48df1f8890aa-kube-api-access-5jbrx\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.764879 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.764776    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4e55def1-38d7-4e6e-a775-48df1f8890aa-cacerts\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.777133 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.777099    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:05.838497 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.838455    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:05.866046 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.866010    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.866627 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.866053    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e55def1-38d7-4e6e-a775-48df1f8890aa-local-certs\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.866627 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.866093    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.866627 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.866117    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jbrx\" (UniqueName: \"kubernetes.io/projected/4e55def1-38d7-4e6e-a775-48df1f8890aa-kube-api-access-5jbrx\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.866627 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.866147    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4e55def1-38d7-4e6e-a775-48df1f8890aa-cacerts\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.866627 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.866198    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-token\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.866627 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.866253    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.866993 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.866890    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.868624 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.868600    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e55def1-38d7-4e6e-a775-48df1f8890aa-local-certs\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.869049 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.869021    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.869142 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.869066    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4e55def1-38d7-4e6e-a775-48df1f8890aa-cacerts\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.869142 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.869099    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.874072 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.874042    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jbrx\" (UniqueName: \"kubernetes.io/projected/4e55def1-38d7-4e6e-a775-48df1f8890aa-kube-api-access-5jbrx\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.874684 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.874661    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-token\") pod \"istiod-openshift-gateway-7574459656-vmvqv\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:05.967242 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:05.967143    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:06.133322 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:06.133266    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"]
Jul 17 21:15:06.134945 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:15:06.134909    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e55def1_38d7_4e6e_a775_48df1f8890aa.slice/crio-32753c8dafb16288a718e82f392e4bd17ec30e2d69682a124801246206739fe5 WatchSource:0}: Error finding container 32753c8dafb16288a718e82f392e4bd17ec30e2d69682a124801246206739fe5: Status 404 returned error can't find the container with id 32753c8dafb16288a718e82f392e4bd17ec30e2d69682a124801246206739fe5
Jul 17 21:15:06.540014 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:06.539895    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-lms8h" event={"ID":"2106c72f-6b22-4c1b-8328-9fb93b711349","Type":"ContainerStarted","Data":"7e615838f3ba6c9cc6d6364f0b50e1990534b93ecc001ee6e8b4d977e897c4da"}
Jul 17 21:15:06.542891 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:06.542838    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv" event={"ID":"4e55def1-38d7-4e6e-a775-48df1f8890aa","Type":"ContainerStarted","Data":"32753c8dafb16288a718e82f392e4bd17ec30e2d69682a124801246206739fe5"}
Jul 17 21:15:06.646042 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:06.645947    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:09.955892 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:09.955862    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 17 21:15:10.571033 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:10.570985    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cf476465c-wbzp5" event={"ID":"efda4d41-9563-4e36-b789-859c4786e297","Type":"ContainerStarted","Data":"2c57bf9e9d924dba6e8fff79a748e36682f4bc84d2a7efb8b8d16791a79276f5"}
Jul 17 21:15:10.573784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:10.573242    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-r8sf5" event={"ID":"aaf6b929-c2d5-48fb-ab6b-535386eb9635","Type":"ContainerStarted","Data":"b52dfbecabe0d13e33c7ce431312ddcbb5c73c77173f739f60db8515a2b7cf84"}
Jul 17 21:15:10.573784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:10.573454    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-r8sf5"
Jul 17 21:15:10.576293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:10.576266    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-lms8h" event={"ID":"2106c72f-6b22-4c1b-8328-9fb93b711349","Type":"ContainerStarted","Data":"6e19703c30ddd1cd462ed2bc850d1ddf86ac70f247d3d1e25da95685be709f65"}
Jul 17 21:15:10.576798 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:10.576777    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-lms8h"
Jul 17 21:15:10.590195 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:10.589986    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7cf476465c-wbzp5" podStartSLOduration=1.950996393 podStartE2EDuration="6.58996707s" totalImagesPullingTime="4.638970677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:15:10.587524354 +0000 UTC m=+424.347091050" watchObservedRunningTime="2026-07-17 21:15:10.58996707 +0000 UTC m=+424.349533765"
Jul 17 21:15:10.603085 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:10.603017    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-r8sf5" podStartSLOduration=2.085040997 podStartE2EDuration="6.602998079s" totalImagesPullingTime="4.517957082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:15:10.601281878 +0000 UTC m=+424.360848573" watchObservedRunningTime="2026-07-17 21:15:10.602998079 +0000 UTC m=+424.362564773"
Jul 17 21:15:10.618011 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:10.617874    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-lms8h" podStartSLOduration=1.324562748 podStartE2EDuration="5.617853376s" totalImagesPullingTime="4.293290628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:15:10.615788339 +0000 UTC m=+424.375355054" watchObservedRunningTime="2026-07-17 21:15:10.617853376 +0000 UTC m=+424.377420069"
Jul 17 21:15:11.036180 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:11.036129    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 21:15:11.036722 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:11.036205    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 17 21:15:11.583935 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:11.583901    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv" event={"ID":"4e55def1-38d7-4e6e-a775-48df1f8890aa","Type":"ContainerStarted","Data":"5cf35519681ce176fd8a3317bc4b703eede4c7867aab621585b1b519f29339cd"}
Jul 17 21:15:11.584386 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:11.584138    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:11.602018 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:11.601965    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv" podStartSLOduration=1.703719827 podStartE2EDuration="6.601950765s" totalImagesPullingTime="4.898230938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:15:11.600253972 +0000 UTC m=+425.359820668" watchObservedRunningTime="2026-07-17 21:15:11.601950765 +0000 UTC m=+425.361517459"
Jul 17 21:15:12.172573 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:12.172532    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:12.272486 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:12.272448    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:12.590153 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:12.590117    2569 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-vmvqv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 21:15:12.590313 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:12.590173    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv" podUID="4e55def1-38d7-4e6e-a775-48df1f8890aa" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 21:15:12.953218 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:12.953130    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:13.148610 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:13.148567    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:13.247029 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:13.246928    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:15.590058 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:15.590024    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:16.212422 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.202544    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:16.286544 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.286499    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-swflf"]
Jul 17 21:15:16.290570 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.290546    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-swflf"
Jul 17 21:15:16.294445 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.294421    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-7crkz\""
Jul 17 21:15:16.294708 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.294685    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 17 21:15:16.296497 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.296468    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l4wzj"]
Jul 17 21:15:16.300114 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.300094    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4wzj"
Jul 17 21:15:16.303517 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.303492    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:16.303766 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.303540    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 17 21:15:16.303904 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.303713    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-tq7vk\""
Jul 17 21:15:16.304221 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.304198    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-swflf"]
Jul 17 21:15:16.326569 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.326539    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l4wzj"]
Jul 17 21:15:16.365422 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.365367    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7134f6e3-e6cc-48f0-85c8-87dd703137d8-cert\") pod \"kserve-controller-manager-d6f4b65b8-swflf\" (UID: \"7134f6e3-e6cc-48f0-85c8-87dd703137d8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-swflf"
Jul 17 21:15:16.365609 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.365447    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4c502b9-5cc5-40d3-ac93-d52a0f629279-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l4wzj\" (UID: \"e4c502b9-5cc5-40d3-ac93-d52a0f629279\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4wzj"
Jul 17 21:15:16.365609 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.365485    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxkqx\" (UniqueName: \"kubernetes.io/projected/e4c502b9-5cc5-40d3-ac93-d52a0f629279-kube-api-access-nxkqx\") pod \"llmisvc-controller-manager-777c6bd7d5-l4wzj\" (UID: \"e4c502b9-5cc5-40d3-ac93-d52a0f629279\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4wzj"
Jul 17 21:15:16.365609 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.365512    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg6vh\" (UniqueName: \"kubernetes.io/projected/7134f6e3-e6cc-48f0-85c8-87dd703137d8-kube-api-access-pg6vh\") pod \"kserve-controller-manager-d6f4b65b8-swflf\" (UID: \"7134f6e3-e6cc-48f0-85c8-87dd703137d8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-swflf"
Jul 17 21:15:16.388203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.388170    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:16.466588 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.466494    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4c502b9-5cc5-40d3-ac93-d52a0f629279-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l4wzj\" (UID: \"e4c502b9-5cc5-40d3-ac93-d52a0f629279\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4wzj"
Jul 17 21:15:16.466588 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.466556    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxkqx\" (UniqueName: \"kubernetes.io/projected/e4c502b9-5cc5-40d3-ac93-d52a0f629279-kube-api-access-nxkqx\") pod \"llmisvc-controller-manager-777c6bd7d5-l4wzj\" (UID: \"e4c502b9-5cc5-40d3-ac93-d52a0f629279\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4wzj"
Jul 17 21:15:16.466588 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.466592    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6vh\" (UniqueName: \"kubernetes.io/projected/7134f6e3-e6cc-48f0-85c8-87dd703137d8-kube-api-access-pg6vh\") pod \"kserve-controller-manager-d6f4b65b8-swflf\" (UID: \"7134f6e3-e6cc-48f0-85c8-87dd703137d8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-swflf"
Jul 17 21:15:16.466873 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.466653    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7134f6e3-e6cc-48f0-85c8-87dd703137d8-cert\") pod \"kserve-controller-manager-d6f4b65b8-swflf\" (UID: \"7134f6e3-e6cc-48f0-85c8-87dd703137d8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-swflf"
Jul 17 21:15:16.466873 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:15:16.466665    2569 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 17 21:15:16.466873 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:15:16.466732    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4c502b9-5cc5-40d3-ac93-d52a0f629279-cert podName:e4c502b9-5cc5-40d3-ac93-d52a0f629279 nodeName:}" failed. No retries permitted until 2026-07-17 21:15:16.966710066 +0000 UTC m=+430.726276738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e4c502b9-5cc5-40d3-ac93-d52a0f629279-cert") pod "llmisvc-controller-manager-777c6bd7d5-l4wzj" (UID: "e4c502b9-5cc5-40d3-ac93-d52a0f629279") : secret "llmisvc-webhook-server-cert" not found
Jul 17 21:15:16.469276 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.469254    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7134f6e3-e6cc-48f0-85c8-87dd703137d8-cert\") pod \"kserve-controller-manager-d6f4b65b8-swflf\" (UID: \"7134f6e3-e6cc-48f0-85c8-87dd703137d8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-swflf"
Jul 17 21:15:16.474902 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.474879    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxkqx\" (UniqueName: \"kubernetes.io/projected/e4c502b9-5cc5-40d3-ac93-d52a0f629279-kube-api-access-nxkqx\") pod \"llmisvc-controller-manager-777c6bd7d5-l4wzj\" (UID: \"e4c502b9-5cc5-40d3-ac93-d52a0f629279\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4wzj"
Jul 17 21:15:16.475203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.475184    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg6vh\" (UniqueName: \"kubernetes.io/projected/7134f6e3-e6cc-48f0-85c8-87dd703137d8-kube-api-access-pg6vh\") pod \"kserve-controller-manager-d6f4b65b8-swflf\" (UID: \"7134f6e3-e6cc-48f0-85c8-87dd703137d8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-swflf"
Jul 17 21:15:16.602254 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.602215    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-swflf"
Jul 17 21:15:16.725067 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.724993    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:16.745342 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.745315    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-swflf"]
Jul 17 21:15:16.970948 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.970908    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4c502b9-5cc5-40d3-ac93-d52a0f629279-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l4wzj\" (UID: \"e4c502b9-5cc5-40d3-ac93-d52a0f629279\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4wzj"
Jul 17 21:15:16.973360 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:16.973333    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4c502b9-5cc5-40d3-ac93-d52a0f629279-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l4wzj\" (UID: \"e4c502b9-5cc5-40d3-ac93-d52a0f629279\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4wzj"
Jul 17 21:15:17.212372 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:17.212336    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4wzj"
Jul 17 21:15:17.351124 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:17.351096    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l4wzj"]
Jul 17 21:15:17.352694 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:15:17.352665    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode4c502b9_5cc5_40d3_ac93_d52a0f629279.slice/crio-ee11c66232bf55582be1e9a0377749c2e22b118c07879a5fd20569183650539f WatchSource:0}: Error finding container ee11c66232bf55582be1e9a0377749c2e22b118c07879a5fd20569183650539f: Status 404 returned error can't find the container with id ee11c66232bf55582be1e9a0377749c2e22b118c07879a5fd20569183650539f
Jul 17 21:15:17.533700 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:17.533615    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:17.613939 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:17.613903    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-swflf" event={"ID":"7134f6e3-e6cc-48f0-85c8-87dd703137d8","Type":"ContainerStarted","Data":"d8d8475063a927a4f3490bd31cd0a3b0c91d2ebb02dc72086514ce454e34147a"}
Jul 17 21:15:17.615048 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:17.615025    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4wzj" event={"ID":"e4c502b9-5cc5-40d3-ac93-d52a0f629279","Type":"ContainerStarted","Data":"ee11c66232bf55582be1e9a0377749c2e22b118c07879a5fd20569183650539f"}
Jul 17 21:15:17.742679 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:17.742643    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:15:20.631427 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:20.631357    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-swflf" event={"ID":"7134f6e3-e6cc-48f0-85c8-87dd703137d8","Type":"ContainerStarted","Data":"aff62fa8a7d779e47020721fc57409cb11fba06c6446c2fee4bab3f764c66e14"}
Jul 17 21:15:20.631886 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:20.631507    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-swflf"
Jul 17 21:15:20.632915 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:20.632891    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4wzj" event={"ID":"e4c502b9-5cc5-40d3-ac93-d52a0f629279","Type":"ContainerStarted","Data":"ed3e074fcbcacad0cdb8ebbcffd97e43408adb63c366f58b11e6977fdf8f5216"}
Jul 17 21:15:20.633042 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:20.633020    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4wzj"
Jul 17 21:15:20.653245 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:20.653193    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-swflf" podStartSLOduration=1.6486513889999999 podStartE2EDuration="4.653178764s" totalImagesPullingTime="3.004527375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:15:20.650932293 +0000 UTC m=+434.410499012" watchObservedRunningTime="2026-07-17 21:15:20.653178764 +0000 UTC m=+434.412745457"
Jul 17 21:15:20.669936 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:20.669882    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4wzj" podStartSLOduration=1.5228895649999998 podStartE2EDuration="4.669865977s" totalImagesPullingTime="3.146976412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:15:20.667977299 +0000 UTC m=+434.427543984" watchObservedRunningTime="2026-07-17 21:15:20.669865977 +0000 UTC m=+434.429432671"
Jul 17 21:15:21.183501 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.183464    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"]
Jul 17 21:15:21.183692 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.183670    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv" podUID="4e55def1-38d7-4e6e-a775-48df1f8890aa" containerName="discovery" containerID="cri-o://5cf35519681ce176fd8a3317bc4b703eede4c7867aab621585b1b519f29339cd" gracePeriod=30
Jul 17 21:15:21.442623 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.442560    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:21.514415 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.514364    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e55def1-38d7-4e6e-a775-48df1f8890aa-local-certs\") pod \"4e55def1-38d7-4e6e-a775-48df1f8890aa\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") "
Jul 17 21:15:21.514415 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.514415    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4e55def1-38d7-4e6e-a775-48df1f8890aa-cacerts\") pod \"4e55def1-38d7-4e6e-a775-48df1f8890aa\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") "
Jul 17 21:15:21.514637 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.514443    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-kubeconfig\") pod \"4e55def1-38d7-4e6e-a775-48df1f8890aa\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") "
Jul 17 21:15:21.514637 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.514490    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-csr-dns-cert\") pod \"4e55def1-38d7-4e6e-a775-48df1f8890aa\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") "
Jul 17 21:15:21.514637 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.514566    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-csr-ca-configmap\") pod \"4e55def1-38d7-4e6e-a775-48df1f8890aa\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") "
Jul 17 21:15:21.514637 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.514614    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-token\") pod \"4e55def1-38d7-4e6e-a775-48df1f8890aa\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") "
Jul 17 21:15:21.514829 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.514700    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jbrx\" (UniqueName: \"kubernetes.io/projected/4e55def1-38d7-4e6e-a775-48df1f8890aa-kube-api-access-5jbrx\") pod \"4e55def1-38d7-4e6e-a775-48df1f8890aa\" (UID: \"4e55def1-38d7-4e6e-a775-48df1f8890aa\") "
Jul 17 21:15:21.514989 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.514957    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "4e55def1-38d7-4e6e-a775-48df1f8890aa" (UID: "4e55def1-38d7-4e6e-a775-48df1f8890aa"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:15:21.517254 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.517132    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-token" (OuterVolumeSpecName: "istio-token") pod "4e55def1-38d7-4e6e-a775-48df1f8890aa" (UID: "4e55def1-38d7-4e6e-a775-48df1f8890aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:15:21.517254 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.517173    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "4e55def1-38d7-4e6e-a775-48df1f8890aa" (UID: "4e55def1-38d7-4e6e-a775-48df1f8890aa"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:15:21.517254 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.517209    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "4e55def1-38d7-4e6e-a775-48df1f8890aa" (UID: "4e55def1-38d7-4e6e-a775-48df1f8890aa"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:15:21.517658 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.517591    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e55def1-38d7-4e6e-a775-48df1f8890aa-local-certs" (OuterVolumeSpecName: "local-certs") pod "4e55def1-38d7-4e6e-a775-48df1f8890aa" (UID: "4e55def1-38d7-4e6e-a775-48df1f8890aa"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:15:21.517658 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.517605    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e55def1-38d7-4e6e-a775-48df1f8890aa-cacerts" (OuterVolumeSpecName: "cacerts") pod "4e55def1-38d7-4e6e-a775-48df1f8890aa" (UID: "4e55def1-38d7-4e6e-a775-48df1f8890aa"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:15:21.517658 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.517611    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e55def1-38d7-4e6e-a775-48df1f8890aa-kube-api-access-5jbrx" (OuterVolumeSpecName: "kube-api-access-5jbrx") pod "4e55def1-38d7-4e6e-a775-48df1f8890aa" (UID: "4e55def1-38d7-4e6e-a775-48df1f8890aa"). InnerVolumeSpecName "kube-api-access-5jbrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:15:21.586793 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.586765    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-r8sf5"
Jul 17 21:15:21.616172 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.616133    2569 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-csr-dns-cert\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:15:21.616172 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.616174    2569 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-csr-ca-configmap\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:15:21.616374 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.616188    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-token\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:15:21.616374 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.616200    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jbrx\" (UniqueName: \"kubernetes.io/projected/4e55def1-38d7-4e6e-a775-48df1f8890aa-kube-api-access-5jbrx\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:15:21.616374 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.616218    2569 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e55def1-38d7-4e6e-a775-48df1f8890aa-local-certs\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:15:21.616374 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.616230    2569 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4e55def1-38d7-4e6e-a775-48df1f8890aa-cacerts\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:15:21.616374 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.616241    2569 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4e55def1-38d7-4e6e-a775-48df1f8890aa-istio-kubeconfig\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:15:21.640036 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.639999    2569 generic.go:356] "Generic (PLEG): container finished" podID="4e55def1-38d7-4e6e-a775-48df1f8890aa" containerID="5cf35519681ce176fd8a3317bc4b703eede4c7867aab621585b1b519f29339cd" exitCode=0
Jul 17 21:15:21.640519 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.640051    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"
Jul 17 21:15:21.640519 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.640051    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv" event={"ID":"4e55def1-38d7-4e6e-a775-48df1f8890aa","Type":"ContainerDied","Data":"5cf35519681ce176fd8a3317bc4b703eede4c7867aab621585b1b519f29339cd"}
Jul 17 21:15:21.640519 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.640124    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv" event={"ID":"4e55def1-38d7-4e6e-a775-48df1f8890aa","Type":"ContainerDied","Data":"32753c8dafb16288a718e82f392e4bd17ec30e2d69682a124801246206739fe5"}
Jul 17 21:15:21.640519 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.640154    2569 scope.go:117] "RemoveContainer" containerID="5cf35519681ce176fd8a3317bc4b703eede4c7867aab621585b1b519f29339cd"
Jul 17 21:15:21.651962 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.651942    2569 scope.go:117] "RemoveContainer" containerID="5cf35519681ce176fd8a3317bc4b703eede4c7867aab621585b1b519f29339cd"
Jul 17 21:15:21.652269 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:15:21.652247    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cf35519681ce176fd8a3317bc4b703eede4c7867aab621585b1b519f29339cd\": container with ID starting with 5cf35519681ce176fd8a3317bc4b703eede4c7867aab621585b1b519f29339cd not found: ID does not exist" containerID="5cf35519681ce176fd8a3317bc4b703eede4c7867aab621585b1b519f29339cd"
Jul 17 21:15:21.652318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.652275    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf35519681ce176fd8a3317bc4b703eede4c7867aab621585b1b519f29339cd"} err="failed to get container status \"5cf35519681ce176fd8a3317bc4b703eede4c7867aab621585b1b519f29339cd\": rpc error: code = NotFound desc = could not find container \"5cf35519681ce176fd8a3317bc4b703eede4c7867aab621585b1b519f29339cd\": container with ID starting with 5cf35519681ce176fd8a3317bc4b703eede4c7867aab621585b1b519f29339cd not found: ID does not exist"
Jul 17 21:15:21.667927 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.667897    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"]
Jul 17 21:15:21.679011 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:21.678980    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vmvqv"]
Jul 17 21:15:22.592232 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:22.592197    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-lms8h"
Jul 17 21:15:22.656632 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:22.656597    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e55def1-38d7-4e6e-a775-48df1f8890aa" path="/var/lib/kubelet/pods/4e55def1-38d7-4e6e-a775-48df1f8890aa/volumes"
Jul 17 21:15:51.642891 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:51.642859    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l4wzj"
Jul 17 21:15:51.645856 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:15:51.645831    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-swflf"
Jul 17 21:16:05.017075 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.017034    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:05.116267 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.116232    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:05.136258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.136223    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bddxt"]
Jul 17 21:16:05.136586 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.136573    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e55def1-38d7-4e6e-a775-48df1f8890aa" containerName="discovery"
Jul 17 21:16:05.136638 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.136588    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e55def1-38d7-4e6e-a775-48df1f8890aa" containerName="discovery"
Jul 17 21:16:05.136672 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.136660    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e55def1-38d7-4e6e-a775-48df1f8890aa" containerName="discovery"
Jul 17 21:16:05.139828 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.139801    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bddxt"
Jul 17 21:16:05.142667 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.142636    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 21:16:05.143064 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.143040    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-nw6jm\""
Jul 17 21:16:05.143064 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.143052    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 21:16:05.150913 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.150889    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bddxt"]
Jul 17 21:16:05.188270 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.188232    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:05.191364 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.191339    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn9gk\" (UniqueName: \"kubernetes.io/projected/2f26a0f6-99b9-497b-aa23-3568d309eb32-kube-api-access-nn9gk\") pod \"kuadrant-operator-catalog-bddxt\" (UID: \"2f26a0f6-99b9-497b-aa23-3568d309eb32\") " pod="kuadrant-system/kuadrant-operator-catalog-bddxt"
Jul 17 21:16:05.292056 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.291972    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn9gk\" (UniqueName: \"kubernetes.io/projected/2f26a0f6-99b9-497b-aa23-3568d309eb32-kube-api-access-nn9gk\") pod \"kuadrant-operator-catalog-bddxt\" (UID: \"2f26a0f6-99b9-497b-aa23-3568d309eb32\") " pod="kuadrant-system/kuadrant-operator-catalog-bddxt"
Jul 17 21:16:05.300730 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.300706    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn9gk\" (UniqueName: \"kubernetes.io/projected/2f26a0f6-99b9-497b-aa23-3568d309eb32-kube-api-access-nn9gk\") pod \"kuadrant-operator-catalog-bddxt\" (UID: \"2f26a0f6-99b9-497b-aa23-3568d309eb32\") " pod="kuadrant-system/kuadrant-operator-catalog-bddxt"
Jul 17 21:16:05.451311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.451280    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bddxt"
Jul 17 21:16:05.507208 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.507151    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bddxt"]
Jul 17 21:16:05.585849 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.585826    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bddxt"]
Jul 17 21:16:05.588248 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:16:05.588217    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f26a0f6_99b9_497b_aa23_3568d309eb32.slice/crio-6df5029d11e1d63e0ad0628cc814af408bc326912b04c19c97a318790c8ef863 WatchSource:0}: Error finding container 6df5029d11e1d63e0ad0628cc814af408bc326912b04c19c97a318790c8ef863: Status 404 returned error can't find the container with id 6df5029d11e1d63e0ad0628cc814af408bc326912b04c19c97a318790c8ef863
Jul 17 21:16:05.715655 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.715615    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rtbw9"]
Jul 17 21:16:05.719988 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.719967    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rtbw9"
Jul 17 21:16:05.726327 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.726302    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rtbw9"]
Jul 17 21:16:05.796933 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.796899    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf9m2\" (UniqueName: \"kubernetes.io/projected/b856ffec-b5f2-4f2a-b3c8-80f28abb5860-kube-api-access-qf9m2\") pod \"kuadrant-operator-catalog-rtbw9\" (UID: \"b856ffec-b5f2-4f2a-b3c8-80f28abb5860\") " pod="kuadrant-system/kuadrant-operator-catalog-rtbw9"
Jul 17 21:16:05.849449 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.849329    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bddxt" event={"ID":"2f26a0f6-99b9-497b-aa23-3568d309eb32","Type":"ContainerStarted","Data":"6df5029d11e1d63e0ad0628cc814af408bc326912b04c19c97a318790c8ef863"}
Jul 17 21:16:05.898277 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.898238    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf9m2\" (UniqueName: \"kubernetes.io/projected/b856ffec-b5f2-4f2a-b3c8-80f28abb5860-kube-api-access-qf9m2\") pod \"kuadrant-operator-catalog-rtbw9\" (UID: \"b856ffec-b5f2-4f2a-b3c8-80f28abb5860\") " pod="kuadrant-system/kuadrant-operator-catalog-rtbw9"
Jul 17 21:16:05.906232 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:05.906200    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf9m2\" (UniqueName: \"kubernetes.io/projected/b856ffec-b5f2-4f2a-b3c8-80f28abb5860-kube-api-access-qf9m2\") pod \"kuadrant-operator-catalog-rtbw9\" (UID: \"b856ffec-b5f2-4f2a-b3c8-80f28abb5860\") " pod="kuadrant-system/kuadrant-operator-catalog-rtbw9"
Jul 17 21:16:06.029748 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:06.029709    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rtbw9"
Jul 17 21:16:06.167755 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:06.167731    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rtbw9"]
Jul 17 21:16:06.169696 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:16:06.169669    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb856ffec_b5f2_4f2a_b3c8_80f28abb5860.slice/crio-8f822d1fb4fdd5cc684d62e257188c2f967c36793cf0dc9859ffd0c1e0e74591 WatchSource:0}: Error finding container 8f822d1fb4fdd5cc684d62e257188c2f967c36793cf0dc9859ffd0c1e0e74591: Status 404 returned error can't find the container with id 8f822d1fb4fdd5cc684d62e257188c2f967c36793cf0dc9859ffd0c1e0e74591
Jul 17 21:16:06.856040 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:06.855996    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rtbw9" event={"ID":"b856ffec-b5f2-4f2a-b3c8-80f28abb5860","Type":"ContainerStarted","Data":"8f822d1fb4fdd5cc684d62e257188c2f967c36793cf0dc9859ffd0c1e0e74591"}
Jul 17 21:16:08.867836 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:08.867777    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bddxt" event={"ID":"2f26a0f6-99b9-497b-aa23-3568d309eb32","Type":"ContainerStarted","Data":"167792a6ddb844dda8716c3fb84173c83a4664559862323333eb2124840b90f4"}
Jul 17 21:16:08.867836 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:08.867801    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-bddxt" podUID="2f26a0f6-99b9-497b-aa23-3568d309eb32" containerName="registry-server" containerID="cri-o://167792a6ddb844dda8716c3fb84173c83a4664559862323333eb2124840b90f4" gracePeriod=2
Jul 17 21:16:08.869408 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:08.869383    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rtbw9" event={"ID":"b856ffec-b5f2-4f2a-b3c8-80f28abb5860","Type":"ContainerStarted","Data":"7104727b37279ab6e0f06e2c2dbe46f88281ec4ac943a64f75b37af3d90a4974"}
Jul 17 21:16:08.882640 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:08.882589    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-bddxt" podStartSLOduration=1.589516266 podStartE2EDuration="3.882573334s" totalImagesPullingTime="2.293057068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:16:08.881952264 +0000 UTC m=+482.641518959" watchObservedRunningTime="2026-07-17 21:16:08.882573334 +0000 UTC m=+482.642140027"
Jul 17 21:16:08.897322 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:08.897280    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-rtbw9" podStartSLOduration=2.184454076 podStartE2EDuration="3.897266983s" totalImagesPullingTime="1.712812907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:16:08.896614862 +0000 UTC m=+482.656181562" watchObservedRunningTime="2026-07-17 21:16:08.897266983 +0000 UTC m=+482.656833677"
Jul 17 21:16:09.121945 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:09.121886    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bddxt"
Jul 17 21:16:09.230135 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:09.230094    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nn9gk\" (UniqueName: \"kubernetes.io/projected/2f26a0f6-99b9-497b-aa23-3568d309eb32-kube-api-access-nn9gk\") pod \"2f26a0f6-99b9-497b-aa23-3568d309eb32\" (UID: \"2f26a0f6-99b9-497b-aa23-3568d309eb32\") "
Jul 17 21:16:09.232349 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:09.232322    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f26a0f6-99b9-497b-aa23-3568d309eb32-kube-api-access-nn9gk" (OuterVolumeSpecName: "kube-api-access-nn9gk") pod "2f26a0f6-99b9-497b-aa23-3568d309eb32" (UID: "2f26a0f6-99b9-497b-aa23-3568d309eb32"). InnerVolumeSpecName "kube-api-access-nn9gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:16:09.331102 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:09.331065    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn9gk\" (UniqueName: \"kubernetes.io/projected/2f26a0f6-99b9-497b-aa23-3568d309eb32-kube-api-access-nn9gk\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:16:09.875278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:09.875245    2569 generic.go:356] "Generic (PLEG): container finished" podID="2f26a0f6-99b9-497b-aa23-3568d309eb32" containerID="167792a6ddb844dda8716c3fb84173c83a4664559862323333eb2124840b90f4" exitCode=0
Jul 17 21:16:09.875698 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:09.875285    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bddxt"
Jul 17 21:16:09.875698 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:09.875335    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bddxt" event={"ID":"2f26a0f6-99b9-497b-aa23-3568d309eb32","Type":"ContainerDied","Data":"167792a6ddb844dda8716c3fb84173c83a4664559862323333eb2124840b90f4"}
Jul 17 21:16:09.875698 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:09.875393    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bddxt" event={"ID":"2f26a0f6-99b9-497b-aa23-3568d309eb32","Type":"ContainerDied","Data":"6df5029d11e1d63e0ad0628cc814af408bc326912b04c19c97a318790c8ef863"}
Jul 17 21:16:09.875698 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:09.875434    2569 scope.go:117] "RemoveContainer" containerID="167792a6ddb844dda8716c3fb84173c83a4664559862323333eb2124840b90f4"
Jul 17 21:16:09.885227 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:09.885211    2569 scope.go:117] "RemoveContainer" containerID="167792a6ddb844dda8716c3fb84173c83a4664559862323333eb2124840b90f4"
Jul 17 21:16:09.885500 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:16:09.885482    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"167792a6ddb844dda8716c3fb84173c83a4664559862323333eb2124840b90f4\": container with ID starting with 167792a6ddb844dda8716c3fb84173c83a4664559862323333eb2124840b90f4 not found: ID does not exist" containerID="167792a6ddb844dda8716c3fb84173c83a4664559862323333eb2124840b90f4"
Jul 17 21:16:09.885578 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:09.885508    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"167792a6ddb844dda8716c3fb84173c83a4664559862323333eb2124840b90f4"} err="failed to get container status \"167792a6ddb844dda8716c3fb84173c83a4664559862323333eb2124840b90f4\": rpc error: code = NotFound desc = could not find container \"167792a6ddb844dda8716c3fb84173c83a4664559862323333eb2124840b90f4\": container with ID starting with 167792a6ddb844dda8716c3fb84173c83a4664559862323333eb2124840b90f4 not found: ID does not exist"
Jul 17 21:16:09.894763 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:09.894740    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bddxt"]
Jul 17 21:16:09.899968 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:09.899948    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bddxt"]
Jul 17 21:16:10.656451 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:10.656384    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f26a0f6-99b9-497b-aa23-3568d309eb32" path="/var/lib/kubelet/pods/2f26a0f6-99b9-497b-aa23-3568d309eb32/volumes"
Jul 17 21:16:11.307796 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:11.307759    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:15.665496 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:15.665458    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:15.745526 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:15.745489    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:16.030756 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:16.030658    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-rtbw9"
Jul 17 21:16:16.030756 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:16.030734    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-rtbw9"
Jul 17 21:16:16.053710 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:16.053684    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-rtbw9"
Jul 17 21:16:16.930508 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:16.930479    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-rtbw9"
Jul 17 21:16:20.495031 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:20.494993    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2"]
Jul 17 21:16:20.495443 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:20.495340    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f26a0f6-99b9-497b-aa23-3568d309eb32" containerName="registry-server"
Jul 17 21:16:20.495443 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:20.495351    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f26a0f6-99b9-497b-aa23-3568d309eb32" containerName="registry-server"
Jul 17 21:16:20.495532 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:20.495473    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f26a0f6-99b9-497b-aa23-3568d309eb32" containerName="registry-server"
Jul 17 21:16:20.499952 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:20.499934    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2"
Jul 17 21:16:20.502129 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:20.502109    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h285x\""
Jul 17 21:16:20.507915 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:20.507892    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2"]
Jul 17 21:16:20.632889 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:20.632845    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0bd5eb95-6b65-4942-b629-5939d4c8e214-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2\" (UID: \"0bd5eb95-6b65-4942-b629-5939d4c8e214\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2"
Jul 17 21:16:20.633063 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:20.632961    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0bd5eb95-6b65-4942-b629-5939d4c8e214-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2\" (UID: \"0bd5eb95-6b65-4942-b629-5939d4c8e214\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2"
Jul 17 21:16:20.633063 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:20.632990    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf6bh\" (UniqueName: \"kubernetes.io/projected/0bd5eb95-6b65-4942-b629-5939d4c8e214-kube-api-access-nf6bh\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2\" (UID: \"0bd5eb95-6b65-4942-b629-5939d4c8e214\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2"
Jul 17 21:16:20.733717 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:20.733671    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0bd5eb95-6b65-4942-b629-5939d4c8e214-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2\" (UID: \"0bd5eb95-6b65-4942-b629-5939d4c8e214\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2"
Jul 17 21:16:20.733717 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:20.733722    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf6bh\" (UniqueName: \"kubernetes.io/projected/0bd5eb95-6b65-4942-b629-5939d4c8e214-kube-api-access-nf6bh\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2\" (UID: \"0bd5eb95-6b65-4942-b629-5939d4c8e214\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2"
Jul 17 21:16:20.733981 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:20.733778    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0bd5eb95-6b65-4942-b629-5939d4c8e214-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2\" (UID: \"0bd5eb95-6b65-4942-b629-5939d4c8e214\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2"
Jul 17 21:16:20.734174 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:20.734153    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0bd5eb95-6b65-4942-b629-5939d4c8e214-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2\" (UID: \"0bd5eb95-6b65-4942-b629-5939d4c8e214\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2"
Jul 17 21:16:20.734247 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:20.734154    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0bd5eb95-6b65-4942-b629-5939d4c8e214-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2\" (UID: \"0bd5eb95-6b65-4942-b629-5939d4c8e214\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2"
Jul 17 21:16:20.743537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:20.743505    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf6bh\" (UniqueName: \"kubernetes.io/projected/0bd5eb95-6b65-4942-b629-5939d4c8e214-kube-api-access-nf6bh\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2\" (UID: \"0bd5eb95-6b65-4942-b629-5939d4c8e214\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2"
Jul 17 21:16:20.810613 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:20.810517    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2"
Jul 17 21:16:20.945705 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:20.945680    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2"]
Jul 17 21:16:20.948087 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:16:20.948059    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd5eb95_6b65_4942_b629_5939d4c8e214.slice/crio-dc7bfa8c82f4ce7d1221cb41dc7414cb8272e51f9c11904a9a28caf1d5c93947 WatchSource:0}: Error finding container dc7bfa8c82f4ce7d1221cb41dc7414cb8272e51f9c11904a9a28caf1d5c93947: Status 404 returned error can't find the container with id dc7bfa8c82f4ce7d1221cb41dc7414cb8272e51f9c11904a9a28caf1d5c93947
Jul 17 21:16:21.092417 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.092373    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b"]
Jul 17 21:16:21.095902 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.095886    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b"
Jul 17 21:16:21.104322 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.104294    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b"]
Jul 17 21:16:21.239569 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.239527    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82253133-dd19-4596-ac01-b7dd1e343483-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b\" (UID: \"82253133-dd19-4596-ac01-b7dd1e343483\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b"
Jul 17 21:16:21.239770 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.239594    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82253133-dd19-4596-ac01-b7dd1e343483-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b\" (UID: \"82253133-dd19-4596-ac01-b7dd1e343483\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b"
Jul 17 21:16:21.239770 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.239718    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkb7b\" (UniqueName: \"kubernetes.io/projected/82253133-dd19-4596-ac01-b7dd1e343483-kube-api-access-lkb7b\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b\" (UID: \"82253133-dd19-4596-ac01-b7dd1e343483\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b"
Jul 17 21:16:21.340966 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.340909    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82253133-dd19-4596-ac01-b7dd1e343483-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b\" (UID: \"82253133-dd19-4596-ac01-b7dd1e343483\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b"
Jul 17 21:16:21.341166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.341014    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkb7b\" (UniqueName: \"kubernetes.io/projected/82253133-dd19-4596-ac01-b7dd1e343483-kube-api-access-lkb7b\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b\" (UID: \"82253133-dd19-4596-ac01-b7dd1e343483\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b"
Jul 17 21:16:21.341166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.341036    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82253133-dd19-4596-ac01-b7dd1e343483-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b\" (UID: \"82253133-dd19-4596-ac01-b7dd1e343483\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b"
Jul 17 21:16:21.341394 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.341372    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82253133-dd19-4596-ac01-b7dd1e343483-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b\" (UID: \"82253133-dd19-4596-ac01-b7dd1e343483\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b"
Jul 17 21:16:21.341394 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.341383    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82253133-dd19-4596-ac01-b7dd1e343483-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b\" (UID: \"82253133-dd19-4596-ac01-b7dd1e343483\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b"
Jul 17 21:16:21.350339 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.350287    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkb7b\" (UniqueName: \"kubernetes.io/projected/82253133-dd19-4596-ac01-b7dd1e343483-kube-api-access-lkb7b\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b\" (UID: \"82253133-dd19-4596-ac01-b7dd1e343483\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b"
Jul 17 21:16:21.414905 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.414868    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b"
Jul 17 21:16:21.546651 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.546626    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b"]
Jul 17 21:16:21.548310 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:16:21.548279    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82253133_dd19_4596_ac01_b7dd1e343483.slice/crio-fd7e252849dece72a6bbc218a3dba2a6112555a8f386aa752fb40f5fb23d3c3f WatchSource:0}: Error finding container fd7e252849dece72a6bbc218a3dba2a6112555a8f386aa752fb40f5fb23d3c3f: Status 404 returned error can't find the container with id fd7e252849dece72a6bbc218a3dba2a6112555a8f386aa752fb40f5fb23d3c3f
Jul 17 21:16:21.692103 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.692066    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb"]
Jul 17 21:16:21.695691 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.695670    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb"
Jul 17 21:16:21.705745 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.705717    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb"]
Jul 17 21:16:21.845693 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.845658    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c82aca7c-ff06-4c79-8c14-198701662c0d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb\" (UID: \"c82aca7c-ff06-4c79-8c14-198701662c0d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb"
Jul 17 21:16:21.845842 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.845726    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb22r\" (UniqueName: \"kubernetes.io/projected/c82aca7c-ff06-4c79-8c14-198701662c0d-kube-api-access-zb22r\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb\" (UID: \"c82aca7c-ff06-4c79-8c14-198701662c0d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb"
Jul 17 21:16:21.845842 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.845763    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c82aca7c-ff06-4c79-8c14-198701662c0d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb\" (UID: \"c82aca7c-ff06-4c79-8c14-198701662c0d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb"
Jul 17 21:16:21.940627 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.940538    2569 generic.go:356] "Generic (PLEG): container finished" podID="0bd5eb95-6b65-4942-b629-5939d4c8e214" containerID="21b578afb4c8eb9865f42a2c9e0cd81707728e53900a88ad54dd60b52b4ff1ee" exitCode=0
Jul 17 21:16:21.940802 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.940620    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2" event={"ID":"0bd5eb95-6b65-4942-b629-5939d4c8e214","Type":"ContainerDied","Data":"21b578afb4c8eb9865f42a2c9e0cd81707728e53900a88ad54dd60b52b4ff1ee"}
Jul 17 21:16:21.940802 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.940669    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2" event={"ID":"0bd5eb95-6b65-4942-b629-5939d4c8e214","Type":"ContainerStarted","Data":"dc7bfa8c82f4ce7d1221cb41dc7414cb8272e51f9c11904a9a28caf1d5c93947"}
Jul 17 21:16:21.942280 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.942257    2569 generic.go:356] "Generic (PLEG): container finished" podID="82253133-dd19-4596-ac01-b7dd1e343483" containerID="89819e58e6fff8291b2c47e0fab822e19224e3e16b207f98c25c6e96c4cbd32c" exitCode=0
Jul 17 21:16:21.942416 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.942343    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b" event={"ID":"82253133-dd19-4596-ac01-b7dd1e343483","Type":"ContainerDied","Data":"89819e58e6fff8291b2c47e0fab822e19224e3e16b207f98c25c6e96c4cbd32c"}
Jul 17 21:16:21.942416 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.942388    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b" event={"ID":"82253133-dd19-4596-ac01-b7dd1e343483","Type":"ContainerStarted","Data":"fd7e252849dece72a6bbc218a3dba2a6112555a8f386aa752fb40f5fb23d3c3f"}
Jul 17 21:16:21.946698 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.946676    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c82aca7c-ff06-4c79-8c14-198701662c0d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb\" (UID: \"c82aca7c-ff06-4c79-8c14-198701662c0d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb"
Jul 17 21:16:21.946824 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.946779    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb22r\" (UniqueName: \"kubernetes.io/projected/c82aca7c-ff06-4c79-8c14-198701662c0d-kube-api-access-zb22r\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb\" (UID: \"c82aca7c-ff06-4c79-8c14-198701662c0d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb"
Jul 17 21:16:21.946886 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.946870    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c82aca7c-ff06-4c79-8c14-198701662c0d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb\" (UID: \"c82aca7c-ff06-4c79-8c14-198701662c0d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb"
Jul 17 21:16:21.947077 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.947058    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c82aca7c-ff06-4c79-8c14-198701662c0d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb\" (UID: \"c82aca7c-ff06-4c79-8c14-198701662c0d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb"
Jul 17 21:16:21.947235 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.947217    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c82aca7c-ff06-4c79-8c14-198701662c0d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb\" (UID: \"c82aca7c-ff06-4c79-8c14-198701662c0d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb"
Jul 17 21:16:21.955430 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:21.955378    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb22r\" (UniqueName: \"kubernetes.io/projected/c82aca7c-ff06-4c79-8c14-198701662c0d-kube-api-access-zb22r\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb\" (UID: \"c82aca7c-ff06-4c79-8c14-198701662c0d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb"
Jul 17 21:16:22.019216 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.019177    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb"
Jul 17 21:16:22.092948 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.092914    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d"]
Jul 17 21:16:22.097890 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.097869    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d"
Jul 17 21:16:22.105871 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.105847    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d"]
Jul 17 21:16:22.161394 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.161367    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb"]
Jul 17 21:16:22.162895 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:16:22.162869    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82aca7c_ff06_4c79_8c14_198701662c0d.slice/crio-1893a9a4e8114966a2d573bf06ae2d7ab81eab256724a05077e359343a914acc WatchSource:0}: Error finding container 1893a9a4e8114966a2d573bf06ae2d7ab81eab256724a05077e359343a914acc: Status 404 returned error can't find the container with id 1893a9a4e8114966a2d573bf06ae2d7ab81eab256724a05077e359343a914acc
Jul 17 21:16:22.249262 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.249213    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d21dbc9-6449-43d8-a58b-40e1333053ec-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d\" (UID: \"3d21dbc9-6449-43d8-a58b-40e1333053ec\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d"
Jul 17 21:16:22.249454 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.249370    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d21dbc9-6449-43d8-a58b-40e1333053ec-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d\" (UID: \"3d21dbc9-6449-43d8-a58b-40e1333053ec\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d"
Jul 17 21:16:22.249454 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.249442    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdlrs\" (UniqueName: \"kubernetes.io/projected/3d21dbc9-6449-43d8-a58b-40e1333053ec-kube-api-access-vdlrs\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d\" (UID: \"3d21dbc9-6449-43d8-a58b-40e1333053ec\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d"
Jul 17 21:16:22.349969 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.349931    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d21dbc9-6449-43d8-a58b-40e1333053ec-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d\" (UID: \"3d21dbc9-6449-43d8-a58b-40e1333053ec\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d"
Jul 17 21:16:22.349969 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.349975    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdlrs\" (UniqueName: \"kubernetes.io/projected/3d21dbc9-6449-43d8-a58b-40e1333053ec-kube-api-access-vdlrs\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d\" (UID: \"3d21dbc9-6449-43d8-a58b-40e1333053ec\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d"
Jul 17 21:16:22.350168 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.350021    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d21dbc9-6449-43d8-a58b-40e1333053ec-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d\" (UID: \"3d21dbc9-6449-43d8-a58b-40e1333053ec\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d"
Jul 17 21:16:22.350347 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.350328    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d21dbc9-6449-43d8-a58b-40e1333053ec-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d\" (UID: \"3d21dbc9-6449-43d8-a58b-40e1333053ec\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d"
Jul 17 21:16:22.350426 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.350384    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d21dbc9-6449-43d8-a58b-40e1333053ec-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d\" (UID: \"3d21dbc9-6449-43d8-a58b-40e1333053ec\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d"
Jul 17 21:16:22.358192 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.358171    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdlrs\" (UniqueName: \"kubernetes.io/projected/3d21dbc9-6449-43d8-a58b-40e1333053ec-kube-api-access-vdlrs\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d\" (UID: \"3d21dbc9-6449-43d8-a58b-40e1333053ec\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d"
Jul 17 21:16:22.419762 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.419727    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d"
Jul 17 21:16:22.555589 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.555559    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d"]
Jul 17 21:16:22.557671 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:16:22.557647    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d21dbc9_6449_43d8_a58b_40e1333053ec.slice/crio-20e08010b46c43edc12a37457ba7bf6858066791a5419287a14ffeb5f66f5700 WatchSource:0}: Error finding container 20e08010b46c43edc12a37457ba7bf6858066791a5419287a14ffeb5f66f5700: Status 404 returned error can't find the container with id 20e08010b46c43edc12a37457ba7bf6858066791a5419287a14ffeb5f66f5700
Jul 17 21:16:22.949210 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.949170    2569 generic.go:356] "Generic (PLEG): container finished" podID="3d21dbc9-6449-43d8-a58b-40e1333053ec" containerID="4a099aef1c6e23fcd85f8a9c132ca49a5c0b436fb89fd94a1b5a7eb9d94eb466" exitCode=0
Jul 17 21:16:22.949388 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.949257    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d" event={"ID":"3d21dbc9-6449-43d8-a58b-40e1333053ec","Type":"ContainerDied","Data":"4a099aef1c6e23fcd85f8a9c132ca49a5c0b436fb89fd94a1b5a7eb9d94eb466"}
Jul 17 21:16:22.949388 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.949313    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d" event={"ID":"3d21dbc9-6449-43d8-a58b-40e1333053ec","Type":"ContainerStarted","Data":"20e08010b46c43edc12a37457ba7bf6858066791a5419287a14ffeb5f66f5700"}
Jul 17 21:16:22.951482 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.951458    2569 generic.go:356] "Generic (PLEG): container finished" podID="0bd5eb95-6b65-4942-b629-5939d4c8e214" containerID="43f880c4e1062f66927887938b37e430dad04d867bef711233094f84c588d650" exitCode=0
Jul 17 21:16:22.951606 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.951547    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2" event={"ID":"0bd5eb95-6b65-4942-b629-5939d4c8e214","Type":"ContainerDied","Data":"43f880c4e1062f66927887938b37e430dad04d867bef711233094f84c588d650"}
Jul 17 21:16:22.953220 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.953196    2569 generic.go:356] "Generic (PLEG): container finished" podID="c82aca7c-ff06-4c79-8c14-198701662c0d" containerID="23ca54fd10ef7bdec7b6ce1239476ecf329c7db5f126fa124cf465c5cec6c297" exitCode=0
Jul 17 21:16:22.953318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.953265    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb" event={"ID":"c82aca7c-ff06-4c79-8c14-198701662c0d","Type":"ContainerDied","Data":"23ca54fd10ef7bdec7b6ce1239476ecf329c7db5f126fa124cf465c5cec6c297"}
Jul 17 21:16:22.953318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:22.953297    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb" event={"ID":"c82aca7c-ff06-4c79-8c14-198701662c0d","Type":"ContainerStarted","Data":"1893a9a4e8114966a2d573bf06ae2d7ab81eab256724a05077e359343a914acc"}
Jul 17 21:16:23.961688 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:23.961662    2569 generic.go:356] "Generic (PLEG): container finished" podID="0bd5eb95-6b65-4942-b629-5939d4c8e214" containerID="da85923481656c177e92d6a8e7d2c37cbe73d99e3e7d1ed87cc67236ca6790d1" exitCode=0
Jul 17 21:16:23.962024 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:23.961774    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2" event={"ID":"0bd5eb95-6b65-4942-b629-5939d4c8e214","Type":"ContainerDied","Data":"da85923481656c177e92d6a8e7d2c37cbe73d99e3e7d1ed87cc67236ca6790d1"}
Jul 17 21:16:23.963924 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:23.963888    2569 generic.go:356] "Generic (PLEG): container finished" podID="82253133-dd19-4596-ac01-b7dd1e343483" containerID="98e3a2d8f670fb31c0b4e7f3d0ff51938942f60a38e123a72c85062076e49e45" exitCode=0
Jul 17 21:16:23.964022 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:23.963922    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b" event={"ID":"82253133-dd19-4596-ac01-b7dd1e343483","Type":"ContainerDied","Data":"98e3a2d8f670fb31c0b4e7f3d0ff51938942f60a38e123a72c85062076e49e45"}
Jul 17 21:16:24.970687 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:24.970648    2569 generic.go:356] "Generic (PLEG): container finished" podID="3d21dbc9-6449-43d8-a58b-40e1333053ec" containerID="4d6d74574011eba2fef6310f465c509979f42ebd49492718fee25ceda5e2ca25" exitCode=0
Jul 17 21:16:24.971111 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:24.970729    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d" event={"ID":"3d21dbc9-6449-43d8-a58b-40e1333053ec","Type":"ContainerDied","Data":"4d6d74574011eba2fef6310f465c509979f42ebd49492718fee25ceda5e2ca25"}
Jul 17 21:16:24.972765 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:24.972745    2569 generic.go:356] "Generic (PLEG): container finished" podID="c82aca7c-ff06-4c79-8c14-198701662c0d" containerID="2e811ef7bf6a81b99c929eb0b8f609c628ebd458c27f628e971debd4b646827b" exitCode=0
Jul 17 21:16:24.972858 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:24.972835    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb" event={"ID":"c82aca7c-ff06-4c79-8c14-198701662c0d","Type":"ContainerDied","Data":"2e811ef7bf6a81b99c929eb0b8f609c628ebd458c27f628e971debd4b646827b"}
Jul 17 21:16:24.975159 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:24.975140    2569 generic.go:356] "Generic (PLEG): container finished" podID="82253133-dd19-4596-ac01-b7dd1e343483" containerID="3e43b6fad3eb738e9dd3ce202d0dc247ab0a9b3e8843aa0dff7f69a5a901144a" exitCode=0
Jul 17 21:16:24.975267 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:24.975222    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b" event={"ID":"82253133-dd19-4596-ac01-b7dd1e343483","Type":"ContainerDied","Data":"3e43b6fad3eb738e9dd3ce202d0dc247ab0a9b3e8843aa0dff7f69a5a901144a"}
Jul 17 21:16:25.123285 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:25.123259    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2"
Jul 17 21:16:25.274322 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:25.274289    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nf6bh\" (UniqueName: \"kubernetes.io/projected/0bd5eb95-6b65-4942-b629-5939d4c8e214-kube-api-access-nf6bh\") pod \"0bd5eb95-6b65-4942-b629-5939d4c8e214\" (UID: \"0bd5eb95-6b65-4942-b629-5939d4c8e214\") "
Jul 17 21:16:25.274486 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:25.274370    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0bd5eb95-6b65-4942-b629-5939d4c8e214-util\") pod \"0bd5eb95-6b65-4942-b629-5939d4c8e214\" (UID: \"0bd5eb95-6b65-4942-b629-5939d4c8e214\") "
Jul 17 21:16:25.274543 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:25.274512    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0bd5eb95-6b65-4942-b629-5939d4c8e214-bundle\") pod \"0bd5eb95-6b65-4942-b629-5939d4c8e214\" (UID: \"0bd5eb95-6b65-4942-b629-5939d4c8e214\") "
Jul 17 21:16:25.274969 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:25.274917    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd5eb95-6b65-4942-b629-5939d4c8e214-bundle" (OuterVolumeSpecName: "bundle") pod "0bd5eb95-6b65-4942-b629-5939d4c8e214" (UID: "0bd5eb95-6b65-4942-b629-5939d4c8e214"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:16:25.276486 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:25.276462    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd5eb95-6b65-4942-b629-5939d4c8e214-kube-api-access-nf6bh" (OuterVolumeSpecName: "kube-api-access-nf6bh") pod "0bd5eb95-6b65-4942-b629-5939d4c8e214" (UID: "0bd5eb95-6b65-4942-b629-5939d4c8e214"). InnerVolumeSpecName "kube-api-access-nf6bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:16:25.279634 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:25.279587    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd5eb95-6b65-4942-b629-5939d4c8e214-util" (OuterVolumeSpecName: "util") pod "0bd5eb95-6b65-4942-b629-5939d4c8e214" (UID: "0bd5eb95-6b65-4942-b629-5939d4c8e214"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:16:25.375918 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:25.375881    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0bd5eb95-6b65-4942-b629-5939d4c8e214-bundle\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:16:25.375918 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:25.375911    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf6bh\" (UniqueName: \"kubernetes.io/projected/0bd5eb95-6b65-4942-b629-5939d4c8e214-kube-api-access-nf6bh\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:16:25.375918 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:25.375922    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0bd5eb95-6b65-4942-b629-5939d4c8e214-util\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:16:25.982633 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:25.982592    2569 generic.go:356] "Generic (PLEG): container finished" podID="3d21dbc9-6449-43d8-a58b-40e1333053ec" containerID="f67ac1ffc8ffcf15f1a2ff149ecddc3b4b462fc6e7bc667658d53bb748a06fc6" exitCode=0
Jul 17 21:16:25.983066 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:25.982675    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d" event={"ID":"3d21dbc9-6449-43d8-a58b-40e1333053ec","Type":"ContainerDied","Data":"f67ac1ffc8ffcf15f1a2ff149ecddc3b4b462fc6e7bc667658d53bb748a06fc6"}
Jul 17 21:16:25.984421 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:25.984381    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2" event={"ID":"0bd5eb95-6b65-4942-b629-5939d4c8e214","Type":"ContainerDied","Data":"dc7bfa8c82f4ce7d1221cb41dc7414cb8272e51f9c11904a9a28caf1d5c93947"}
Jul 17 21:16:25.984421 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:25.984396    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2"
Jul 17 21:16:25.984561 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:25.984440    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc7bfa8c82f4ce7d1221cb41dc7414cb8272e51f9c11904a9a28caf1d5c93947"
Jul 17 21:16:25.986508 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:25.986485    2569 generic.go:356] "Generic (PLEG): container finished" podID="c82aca7c-ff06-4c79-8c14-198701662c0d" containerID="4bb9a740b0b844c0dc0ce14176f6d570ee7379c7b02fa2a5592c09abd977152a" exitCode=0
Jul 17 21:16:25.986648 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:25.986525    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb" event={"ID":"c82aca7c-ff06-4c79-8c14-198701662c0d","Type":"ContainerDied","Data":"4bb9a740b0b844c0dc0ce14176f6d570ee7379c7b02fa2a5592c09abd977152a"}
Jul 17 21:16:26.130690 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:26.130661    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b"
Jul 17 21:16:26.283832 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:26.283774    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82253133-dd19-4596-ac01-b7dd1e343483-util\") pod \"82253133-dd19-4596-ac01-b7dd1e343483\" (UID: \"82253133-dd19-4596-ac01-b7dd1e343483\") "
Jul 17 21:16:26.283832 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:26.283844    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82253133-dd19-4596-ac01-b7dd1e343483-bundle\") pod \"82253133-dd19-4596-ac01-b7dd1e343483\" (UID: \"82253133-dd19-4596-ac01-b7dd1e343483\") "
Jul 17 21:16:26.284091 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:26.283910    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkb7b\" (UniqueName: \"kubernetes.io/projected/82253133-dd19-4596-ac01-b7dd1e343483-kube-api-access-lkb7b\") pod \"82253133-dd19-4596-ac01-b7dd1e343483\" (UID: \"82253133-dd19-4596-ac01-b7dd1e343483\") "
Jul 17 21:16:26.284557 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:26.284526    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82253133-dd19-4596-ac01-b7dd1e343483-bundle" (OuterVolumeSpecName: "bundle") pod "82253133-dd19-4596-ac01-b7dd1e343483" (UID: "82253133-dd19-4596-ac01-b7dd1e343483"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:16:26.286016 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:26.285992    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82253133-dd19-4596-ac01-b7dd1e343483-kube-api-access-lkb7b" (OuterVolumeSpecName: "kube-api-access-lkb7b") pod "82253133-dd19-4596-ac01-b7dd1e343483" (UID: "82253133-dd19-4596-ac01-b7dd1e343483"). InnerVolumeSpecName "kube-api-access-lkb7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:16:26.289198 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:26.289164    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82253133-dd19-4596-ac01-b7dd1e343483-util" (OuterVolumeSpecName: "util") pod "82253133-dd19-4596-ac01-b7dd1e343483" (UID: "82253133-dd19-4596-ac01-b7dd1e343483"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:16:26.384608 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:26.384573    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82253133-dd19-4596-ac01-b7dd1e343483-util\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:16:26.384608 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:26.384603    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82253133-dd19-4596-ac01-b7dd1e343483-bundle\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:16:26.384844 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:26.384617    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkb7b\" (UniqueName: \"kubernetes.io/projected/82253133-dd19-4596-ac01-b7dd1e343483-kube-api-access-lkb7b\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:16:26.500319 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:26.500289    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:26.557706 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:26.557676    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:26.612704 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:26.612668    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:26.993491 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:26.993410    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b"
Jul 17 21:16:26.993491 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:26.993440    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b" event={"ID":"82253133-dd19-4596-ac01-b7dd1e343483","Type":"ContainerDied","Data":"fd7e252849dece72a6bbc218a3dba2a6112555a8f386aa752fb40f5fb23d3c3f"}
Jul 17 21:16:26.993904 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:26.993492    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd7e252849dece72a6bbc218a3dba2a6112555a8f386aa752fb40f5fb23d3c3f"
Jul 17 21:16:27.152195 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:27.152170    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb"
Jul 17 21:16:27.154291 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:27.154269    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d"
Jul 17 21:16:27.293271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:27.293186    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d21dbc9-6449-43d8-a58b-40e1333053ec-util\") pod \"3d21dbc9-6449-43d8-a58b-40e1333053ec\" (UID: \"3d21dbc9-6449-43d8-a58b-40e1333053ec\") "
Jul 17 21:16:27.293271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:27.293225    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c82aca7c-ff06-4c79-8c14-198701662c0d-bundle\") pod \"c82aca7c-ff06-4c79-8c14-198701662c0d\" (UID: \"c82aca7c-ff06-4c79-8c14-198701662c0d\") "
Jul 17 21:16:27.293271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:27.293247    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb22r\" (UniqueName: \"kubernetes.io/projected/c82aca7c-ff06-4c79-8c14-198701662c0d-kube-api-access-zb22r\") pod \"c82aca7c-ff06-4c79-8c14-198701662c0d\" (UID: \"c82aca7c-ff06-4c79-8c14-198701662c0d\") "
Jul 17 21:16:27.293271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:27.293266    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d21dbc9-6449-43d8-a58b-40e1333053ec-bundle\") pod \"3d21dbc9-6449-43d8-a58b-40e1333053ec\" (UID: \"3d21dbc9-6449-43d8-a58b-40e1333053ec\") "
Jul 17 21:16:27.293653 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:27.293317    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdlrs\" (UniqueName: \"kubernetes.io/projected/3d21dbc9-6449-43d8-a58b-40e1333053ec-kube-api-access-vdlrs\") pod \"3d21dbc9-6449-43d8-a58b-40e1333053ec\" (UID: \"3d21dbc9-6449-43d8-a58b-40e1333053ec\") "
Jul 17 21:16:27.293653 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:27.293394    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c82aca7c-ff06-4c79-8c14-198701662c0d-util\") pod \"c82aca7c-ff06-4c79-8c14-198701662c0d\" (UID: \"c82aca7c-ff06-4c79-8c14-198701662c0d\") "
Jul 17 21:16:27.293834 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:27.293811    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c82aca7c-ff06-4c79-8c14-198701662c0d-bundle" (OuterVolumeSpecName: "bundle") pod "c82aca7c-ff06-4c79-8c14-198701662c0d" (UID: "c82aca7c-ff06-4c79-8c14-198701662c0d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:16:27.293925 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:27.293900    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d21dbc9-6449-43d8-a58b-40e1333053ec-bundle" (OuterVolumeSpecName: "bundle") pod "3d21dbc9-6449-43d8-a58b-40e1333053ec" (UID: "3d21dbc9-6449-43d8-a58b-40e1333053ec"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:16:27.296025 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:27.295991    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82aca7c-ff06-4c79-8c14-198701662c0d-kube-api-access-zb22r" (OuterVolumeSpecName: "kube-api-access-zb22r") pod "c82aca7c-ff06-4c79-8c14-198701662c0d" (UID: "c82aca7c-ff06-4c79-8c14-198701662c0d"). InnerVolumeSpecName "kube-api-access-zb22r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:16:27.296110 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:27.296031    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d21dbc9-6449-43d8-a58b-40e1333053ec-kube-api-access-vdlrs" (OuterVolumeSpecName: "kube-api-access-vdlrs") pod "3d21dbc9-6449-43d8-a58b-40e1333053ec" (UID: "3d21dbc9-6449-43d8-a58b-40e1333053ec"). InnerVolumeSpecName "kube-api-access-vdlrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:16:27.299051 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:27.299032    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d21dbc9-6449-43d8-a58b-40e1333053ec-util" (OuterVolumeSpecName: "util") pod "3d21dbc9-6449-43d8-a58b-40e1333053ec" (UID: "3d21dbc9-6449-43d8-a58b-40e1333053ec"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:16:27.364749 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:27.364686    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c82aca7c-ff06-4c79-8c14-198701662c0d-util" (OuterVolumeSpecName: "util") pod "c82aca7c-ff06-4c79-8c14-198701662c0d" (UID: "c82aca7c-ff06-4c79-8c14-198701662c0d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:16:27.394690 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:27.394651    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d21dbc9-6449-43d8-a58b-40e1333053ec-bundle\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:16:27.394690 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:27.394684    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdlrs\" (UniqueName: \"kubernetes.io/projected/3d21dbc9-6449-43d8-a58b-40e1333053ec-kube-api-access-vdlrs\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:16:27.394690 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:27.394695    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c82aca7c-ff06-4c79-8c14-198701662c0d-util\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:16:27.394924 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:27.394704    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d21dbc9-6449-43d8-a58b-40e1333053ec-util\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:16:27.394924 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:27.394713    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c82aca7c-ff06-4c79-8c14-198701662c0d-bundle\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:16:27.394924 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:27.394721    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb22r\" (UniqueName: \"kubernetes.io/projected/c82aca7c-ff06-4c79-8c14-198701662c0d-kube-api-access-zb22r\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:16:28.001036 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:28.001008    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb"
Jul 17 21:16:28.001514 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:28.001036    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb" event={"ID":"c82aca7c-ff06-4c79-8c14-198701662c0d","Type":"ContainerDied","Data":"1893a9a4e8114966a2d573bf06ae2d7ab81eab256724a05077e359343a914acc"}
Jul 17 21:16:28.001514 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:28.001089    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1893a9a4e8114966a2d573bf06ae2d7ab81eab256724a05077e359343a914acc"
Jul 17 21:16:28.003104 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:28.003075    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d" event={"ID":"3d21dbc9-6449-43d8-a58b-40e1333053ec","Type":"ContainerDied","Data":"20e08010b46c43edc12a37457ba7bf6858066791a5419287a14ffeb5f66f5700"}
Jul 17 21:16:28.003219 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:28.003108    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d"
Jul 17 21:16:28.003219 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:28.003124    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20e08010b46c43edc12a37457ba7bf6858066791a5419287a14ffeb5f66f5700"
Jul 17 21:16:34.266270 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:34.266223    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:36.902501 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902463    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-b9bnv"]
Jul 17 21:16:36.902881 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902807    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bd5eb95-6b65-4942-b629-5939d4c8e214" containerName="util"
Jul 17 21:16:36.902881 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902818    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bd5eb95-6b65-4942-b629-5939d4c8e214" containerName="util"
Jul 17 21:16:36.902881 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902828    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82aca7c-ff06-4c79-8c14-198701662c0d" containerName="util"
Jul 17 21:16:36.902881 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902834    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82aca7c-ff06-4c79-8c14-198701662c0d" containerName="util"
Jul 17 21:16:36.902881 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902840    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bd5eb95-6b65-4942-b629-5939d4c8e214" containerName="extract"
Jul 17 21:16:36.902881 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902847    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bd5eb95-6b65-4942-b629-5939d4c8e214" containerName="extract"
Jul 17 21:16:36.902881 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902853    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d21dbc9-6449-43d8-a58b-40e1333053ec" containerName="pull"
Jul 17 21:16:36.902881 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902858    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d21dbc9-6449-43d8-a58b-40e1333053ec" containerName="pull"
Jul 17 21:16:36.902881 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902871    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82aca7c-ff06-4c79-8c14-198701662c0d" containerName="pull"
Jul 17 21:16:36.902881 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902876    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82aca7c-ff06-4c79-8c14-198701662c0d" containerName="pull"
Jul 17 21:16:36.902881 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902884    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82253133-dd19-4596-ac01-b7dd1e343483" containerName="pull"
Jul 17 21:16:36.902881 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902889    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="82253133-dd19-4596-ac01-b7dd1e343483" containerName="pull"
Jul 17 21:16:36.903258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902896    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bd5eb95-6b65-4942-b629-5939d4c8e214" containerName="pull"
Jul 17 21:16:36.903258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902901    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bd5eb95-6b65-4942-b629-5939d4c8e214" containerName="pull"
Jul 17 21:16:36.903258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902905    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82253133-dd19-4596-ac01-b7dd1e343483" containerName="extract"
Jul 17 21:16:36.903258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902911    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="82253133-dd19-4596-ac01-b7dd1e343483" containerName="extract"
Jul 17 21:16:36.903258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902916    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d21dbc9-6449-43d8-a58b-40e1333053ec" containerName="extract"
Jul 17 21:16:36.903258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902921    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d21dbc9-6449-43d8-a58b-40e1333053ec" containerName="extract"
Jul 17 21:16:36.903258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902927    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d21dbc9-6449-43d8-a58b-40e1333053ec" containerName="util"
Jul 17 21:16:36.903258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902931    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d21dbc9-6449-43d8-a58b-40e1333053ec" containerName="util"
Jul 17 21:16:36.903258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902939    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82253133-dd19-4596-ac01-b7dd1e343483" containerName="util"
Jul 17 21:16:36.903258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902943    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="82253133-dd19-4596-ac01-b7dd1e343483" containerName="util"
Jul 17 21:16:36.903258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902950    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82aca7c-ff06-4c79-8c14-198701662c0d" containerName="extract"
Jul 17 21:16:36.903258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.902957    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82aca7c-ff06-4c79-8c14-198701662c0d" containerName="extract"
Jul 17 21:16:36.903258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.903028    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d21dbc9-6449-43d8-a58b-40e1333053ec" containerName="extract"
Jul 17 21:16:36.903258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.903035    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="82253133-dd19-4596-ac01-b7dd1e343483" containerName="extract"
Jul 17 21:16:36.903258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.903043    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c82aca7c-ff06-4c79-8c14-198701662c0d" containerName="extract"
Jul 17 21:16:36.903258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.903050    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bd5eb95-6b65-4942-b629-5939d4c8e214" containerName="extract"
Jul 17 21:16:36.906144 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.906126    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b9bnv"
Jul 17 21:16:36.908207 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.908176    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 17 21:16:36.908207 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.908176    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-j8pnh\""
Jul 17 21:16:36.917970 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.917947    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-b9bnv"]
Jul 17 21:16:36.974087 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:36.974050    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsvn2\" (UniqueName: \"kubernetes.io/projected/a7a6ffaf-914c-475b-9acc-f07241caca1d-kube-api-access-hsvn2\") pod \"dns-operator-controller-manager-864dd78c4c-b9bnv\" (UID: \"a7a6ffaf-914c-475b-9acc-f07241caca1d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b9bnv"
Jul 17 21:16:37.075167 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:37.075128    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsvn2\" (UniqueName: \"kubernetes.io/projected/a7a6ffaf-914c-475b-9acc-f07241caca1d-kube-api-access-hsvn2\") pod \"dns-operator-controller-manager-864dd78c4c-b9bnv\" (UID: \"a7a6ffaf-914c-475b-9acc-f07241caca1d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b9bnv"
Jul 17 21:16:37.089034 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:37.089009    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsvn2\" (UniqueName: \"kubernetes.io/projected/a7a6ffaf-914c-475b-9acc-f07241caca1d-kube-api-access-hsvn2\") pod \"dns-operator-controller-manager-864dd78c4c-b9bnv\" (UID: \"a7a6ffaf-914c-475b-9acc-f07241caca1d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b9bnv"
Jul 17 21:16:37.217952 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:37.217869    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b9bnv"
Jul 17 21:16:37.408618 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:37.408581    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-b9bnv"]
Jul 17 21:16:37.412022 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:16:37.411994    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a6ffaf_914c_475b_9acc_f07241caca1d.slice/crio-16623e6ca2c81d651f818f94ae625083f96d412a8f1e3c05ea423e06cb34a96f WatchSource:0}: Error finding container 16623e6ca2c81d651f818f94ae625083f96d412a8f1e3c05ea423e06cb34a96f: Status 404 returned error can't find the container with id 16623e6ca2c81d651f818f94ae625083f96d412a8f1e3c05ea423e06cb34a96f
Jul 17 21:16:37.867367 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:37.867331    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:38.056485 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:38.056450    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b9bnv" event={"ID":"a7a6ffaf-914c-475b-9acc-f07241caca1d","Type":"ContainerStarted","Data":"16623e6ca2c81d651f818f94ae625083f96d412a8f1e3c05ea423e06cb34a96f"}
Jul 17 21:16:40.069758 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:40.069720    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b9bnv" event={"ID":"a7a6ffaf-914c-475b-9acc-f07241caca1d","Type":"ContainerStarted","Data":"e3819d811841a8851410ca3ce47053c43515c614e3bdb9f526226c7487d6bba7"}
Jul 17 21:16:40.070144 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:40.069846    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b9bnv"
Jul 17 21:16:40.089032 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:40.088987    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b9bnv" podStartSLOduration=2.222665824 podStartE2EDuration="4.088972319s" totalImagesPullingTime="1.866306495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:16:40.087600757 +0000 UTC m=+513.847167451" watchObservedRunningTime="2026-07-17 21:16:40.088972319 +0000 UTC m=+513.848539083"
Jul 17 21:16:40.972956 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:40.972916    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:42.059344 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:42.059308    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"]
Jul 17 21:16:42.067044 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:42.067022    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"
Jul 17 21:16:42.070953 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:42.070927    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tp7kt\""
Jul 17 21:16:42.089067 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:42.089031    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"]
Jul 17 21:16:42.110910 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:42.110878    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckwq8\" (UniqueName: \"kubernetes.io/projected/94ee4c30-ebb4-4671-83bc-aa54e4546cdd-kube-api-access-ckwq8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wspbh\" (UID: \"94ee4c30-ebb4-4671-83bc-aa54e4546cdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"
Jul 17 21:16:42.111076 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:42.110959    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94ee4c30-ebb4-4671-83bc-aa54e4546cdd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wspbh\" (UID: \"94ee4c30-ebb4-4671-83bc-aa54e4546cdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"
Jul 17 21:16:42.152750 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:42.152702    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:42.211914 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:42.211879    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckwq8\" (UniqueName: \"kubernetes.io/projected/94ee4c30-ebb4-4671-83bc-aa54e4546cdd-kube-api-access-ckwq8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wspbh\" (UID: \"94ee4c30-ebb4-4671-83bc-aa54e4546cdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"
Jul 17 21:16:42.212103 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:42.211966    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94ee4c30-ebb4-4671-83bc-aa54e4546cdd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wspbh\" (UID: \"94ee4c30-ebb4-4671-83bc-aa54e4546cdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"
Jul 17 21:16:42.212329 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:42.212312    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94ee4c30-ebb4-4671-83bc-aa54e4546cdd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wspbh\" (UID: \"94ee4c30-ebb4-4671-83bc-aa54e4546cdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"
Jul 17 21:16:42.223991 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:42.223958    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckwq8\" (UniqueName: \"kubernetes.io/projected/94ee4c30-ebb4-4671-83bc-aa54e4546cdd-kube-api-access-ckwq8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wspbh\" (UID: \"94ee4c30-ebb4-4671-83bc-aa54e4546cdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"
Jul 17 21:16:42.378941 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:42.378905    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"
Jul 17 21:16:42.474201 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:42.474163    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:42.523826 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:42.523800    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"]
Jul 17 21:16:42.525669 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:16:42.525637    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ee4c30_ebb4_4671_83bc_aa54e4546cdd.slice/crio-f33d321de7b9723416427a576934b058b8e77eaf55b018c0eb8d764d48aa5b95 WatchSource:0}: Error finding container f33d321de7b9723416427a576934b058b8e77eaf55b018c0eb8d764d48aa5b95: Status 404 returned error can't find the container with id f33d321de7b9723416427a576934b058b8e77eaf55b018c0eb8d764d48aa5b95
Jul 17 21:16:43.086080 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:43.086046    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh" event={"ID":"94ee4c30-ebb4-4671-83bc-aa54e4546cdd","Type":"ContainerStarted","Data":"f33d321de7b9723416427a576934b058b8e77eaf55b018c0eb8d764d48aa5b95"}
Jul 17 21:16:43.264921 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:43.264869    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:45.935195 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:45.935117    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2m5tx"]
Jul 17 21:16:45.938680 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:45.938658    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2m5tx"
Jul 17 21:16:45.940752 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:45.940733    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-l2hfx\""
Jul 17 21:16:45.945045 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:45.945019    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk7d9\" (UniqueName: \"kubernetes.io/projected/07cffc65-fcdd-49b3-922f-c5af96a7070e-kube-api-access-kk7d9\") pod \"authorino-operator-75cc8d5cb7-2m5tx\" (UID: \"07cffc65-fcdd-49b3-922f-c5af96a7070e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2m5tx"
Jul 17 21:16:45.950866 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:45.950843    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:45.951297 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:45.951279    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2m5tx"]
Jul 17 21:16:46.046272 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:46.046232    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk7d9\" (UniqueName: \"kubernetes.io/projected/07cffc65-fcdd-49b3-922f-c5af96a7070e-kube-api-access-kk7d9\") pod \"authorino-operator-75cc8d5cb7-2m5tx\" (UID: \"07cffc65-fcdd-49b3-922f-c5af96a7070e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2m5tx"
Jul 17 21:16:46.057718 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:46.057683    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk7d9\" (UniqueName: \"kubernetes.io/projected/07cffc65-fcdd-49b3-922f-c5af96a7070e-kube-api-access-kk7d9\") pod \"authorino-operator-75cc8d5cb7-2m5tx\" (UID: \"07cffc65-fcdd-49b3-922f-c5af96a7070e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2m5tx"
Jul 17 21:16:46.250691 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:46.250604    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2m5tx"
Jul 17 21:16:46.410169 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:46.410032    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2m5tx"]
Jul 17 21:16:46.410551 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:16:46.410522    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07cffc65_fcdd_49b3_922f_c5af96a7070e.slice/crio-55415c1994bfde992618f98a5f6390dd61d4891c93609bbe986320c30cb7187f WatchSource:0}: Error finding container 55415c1994bfde992618f98a5f6390dd61d4891c93609bbe986320c30cb7187f: Status 404 returned error can't find the container with id 55415c1994bfde992618f98a5f6390dd61d4891c93609bbe986320c30cb7187f
Jul 17 21:16:46.867547 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:46.867495    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:47.112936 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:47.112892    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2m5tx" event={"ID":"07cffc65-fcdd-49b3-922f-c5af96a7070e","Type":"ContainerStarted","Data":"55415c1994bfde992618f98a5f6390dd61d4891c93609bbe986320c30cb7187f"}
Jul 17 21:16:48.781586 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:48.781543    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:48.831982 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:48.831945    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"]
Jul 17 21:16:48.835078 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:48.835055    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"
Jul 17 21:16:48.837095 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:48.837076    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-d8gfw\""
Jul 17 21:16:48.846711 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:48.846684    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"]
Jul 17 21:16:48.872206 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:48.872172    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf6j4\" (UniqueName: \"kubernetes.io/projected/72cac949-57ff-49b6-82b7-a4fe74884ac5-kube-api-access-kf6j4\") pod \"limitador-operator-controller-manager-859d7b78bd-4z9dp\" (UID: \"72cac949-57ff-49b6-82b7-a4fe74884ac5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"
Jul 17 21:16:48.973337 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:48.973301    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf6j4\" (UniqueName: \"kubernetes.io/projected/72cac949-57ff-49b6-82b7-a4fe74884ac5-kube-api-access-kf6j4\") pod \"limitador-operator-controller-manager-859d7b78bd-4z9dp\" (UID: \"72cac949-57ff-49b6-82b7-a4fe74884ac5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"
Jul 17 21:16:48.987841 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:48.987809    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf6j4\" (UniqueName: \"kubernetes.io/projected/72cac949-57ff-49b6-82b7-a4fe74884ac5-kube-api-access-kf6j4\") pod \"limitador-operator-controller-manager-859d7b78bd-4z9dp\" (UID: \"72cac949-57ff-49b6-82b7-a4fe74884ac5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"
Jul 17 21:16:49.147956 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:49.147925    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"
Jul 17 21:16:49.950926 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:49.950902    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"]
Jul 17 21:16:49.952628 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:16:49.952601    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72cac949_57ff_49b6_82b7_a4fe74884ac5.slice/crio-b7f1f5a012ad8b126ceff2b7b7497b2bc5b06abd84e9ed48960ddc86e098916c WatchSource:0}: Error finding container b7f1f5a012ad8b126ceff2b7b7497b2bc5b06abd84e9ed48960ddc86e098916c: Status 404 returned error can't find the container with id b7f1f5a012ad8b126ceff2b7b7497b2bc5b06abd84e9ed48960ddc86e098916c
Jul 17 21:16:50.131518 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:50.131471    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp" event={"ID":"72cac949-57ff-49b6-82b7-a4fe74884ac5","Type":"ContainerStarted","Data":"b7f1f5a012ad8b126ceff2b7b7497b2bc5b06abd84e9ed48960ddc86e098916c"}
Jul 17 21:16:50.132960 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:50.132920    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh" event={"ID":"94ee4c30-ebb4-4671-83bc-aa54e4546cdd","Type":"ContainerStarted","Data":"c3eb9cd332da8fd946c65fcaf01623938b840aac51a2761216a22b41320633dc"}
Jul 17 21:16:50.133120 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:50.133015    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"
Jul 17 21:16:50.134417 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:50.134383    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2m5tx" event={"ID":"07cffc65-fcdd-49b3-922f-c5af96a7070e","Type":"ContainerStarted","Data":"6d82b604fb8bf905fa20e6827f7131cfc592402ebd84dcb5fe150ba6af018baf"}
Jul 17 21:16:50.134529 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:50.134514    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2m5tx"
Jul 17 21:16:50.154442 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:50.154373    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh" podStartSLOduration=0.835031567 podStartE2EDuration="8.154362177s" totalImagesPullingTime="7.31933061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:16:50.154199889 +0000 UTC m=+523.913766583" watchObservedRunningTime="2026-07-17 21:16:50.154362177 +0000 UTC m=+523.913928871"
Jul 17 21:16:50.171270 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:50.171223    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2m5tx" podStartSLOduration=1.725845868 podStartE2EDuration="5.171205381s" totalImagesPullingTime="3.445359513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:16:50.170431507 +0000 UTC m=+523.929998201" watchObservedRunningTime="2026-07-17 21:16:50.171205381 +0000 UTC m=+523.930772076"
Jul 17 21:16:50.279801 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:50.279758    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:50.380379 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:50.380339    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:51.077144 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:51.077115    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b9bnv"
Jul 17 21:16:52.147736 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:52.147700    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp" event={"ID":"72cac949-57ff-49b6-82b7-a4fe74884ac5","Type":"ContainerStarted","Data":"87e461cf8be4b47624b8b970ff640a9ff73c5153f65d9426b6152795ad3ac45b"}
Jul 17 21:16:52.148112 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:52.147797    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"
Jul 17 21:16:52.198447 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:52.198380    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp" podStartSLOduration=2.525104717 podStartE2EDuration="4.198362095s" totalImagesPullingTime="1.673257378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:16:52.195824084 +0000 UTC m=+525.955390779" watchObservedRunningTime="2026-07-17 21:16:52.198362095 +0000 UTC m=+525.957928789"
Jul 17 21:16:52.733452 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:52.733414    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dljpp"]
Jul 17 21:16:52.736445 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:52.736395    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:52.737300 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:52.737279    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dljpp"
Jul 17 21:16:52.739627 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:52.739551    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h285x\""
Jul 17 21:16:52.739881 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:52.739863    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 17 21:16:52.740239 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:52.740206    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 17 21:16:52.745229 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:52.745205    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dljpp"]
Jul 17 21:16:52.807617 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:52.807571    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57dbp\" (UniqueName: \"kubernetes.io/projected/a09d35c4-d15f-45ad-855f-1b43f8fac1bf-kube-api-access-57dbp\") pod \"kuadrant-console-plugin-66b5774985-dljpp\" (UID: \"a09d35c4-d15f-45ad-855f-1b43f8fac1bf\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dljpp"
Jul 17 21:16:52.807807 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:52.807634    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a09d35c4-d15f-45ad-855f-1b43f8fac1bf-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dljpp\" (UID: \"a09d35c4-d15f-45ad-855f-1b43f8fac1bf\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dljpp"
Jul 17 21:16:52.807807 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:52.807691    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a09d35c4-d15f-45ad-855f-1b43f8fac1bf-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dljpp\" (UID: \"a09d35c4-d15f-45ad-855f-1b43f8fac1bf\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dljpp"
Jul 17 21:16:52.908971 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:52.908929    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57dbp\" (UniqueName: \"kubernetes.io/projected/a09d35c4-d15f-45ad-855f-1b43f8fac1bf-kube-api-access-57dbp\") pod \"kuadrant-console-plugin-66b5774985-dljpp\" (UID: \"a09d35c4-d15f-45ad-855f-1b43f8fac1bf\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dljpp"
Jul 17 21:16:52.909185 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:52.908988    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a09d35c4-d15f-45ad-855f-1b43f8fac1bf-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dljpp\" (UID: \"a09d35c4-d15f-45ad-855f-1b43f8fac1bf\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dljpp"
Jul 17 21:16:52.909185 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:16:52.909083    2569 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 17 21:16:52.909185 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:16:52.909135    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a09d35c4-d15f-45ad-855f-1b43f8fac1bf-plugin-serving-cert podName:a09d35c4-d15f-45ad-855f-1b43f8fac1bf nodeName:}" failed. No retries permitted until 2026-07-17 21:16:53.409118839 +0000 UTC m=+527.168685510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/a09d35c4-d15f-45ad-855f-1b43f8fac1bf-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-dljpp" (UID: "a09d35c4-d15f-45ad-855f-1b43f8fac1bf") : secret "plugin-serving-cert" not found
Jul 17 21:16:52.909185 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:52.909165    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a09d35c4-d15f-45ad-855f-1b43f8fac1bf-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dljpp\" (UID: \"a09d35c4-d15f-45ad-855f-1b43f8fac1bf\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dljpp"
Jul 17 21:16:52.909810 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:52.909791    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a09d35c4-d15f-45ad-855f-1b43f8fac1bf-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dljpp\" (UID: \"a09d35c4-d15f-45ad-855f-1b43f8fac1bf\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dljpp"
Jul 17 21:16:52.917067 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:52.917039    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57dbp\" (UniqueName: \"kubernetes.io/projected/a09d35c4-d15f-45ad-855f-1b43f8fac1bf-kube-api-access-57dbp\") pod \"kuadrant-console-plugin-66b5774985-dljpp\" (UID: \"a09d35c4-d15f-45ad-855f-1b43f8fac1bf\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dljpp"
Jul 17 21:16:53.045753 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:53.045658    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:53.413231 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:53.413192    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a09d35c4-d15f-45ad-855f-1b43f8fac1bf-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dljpp\" (UID: \"a09d35c4-d15f-45ad-855f-1b43f8fac1bf\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dljpp"
Jul 17 21:16:53.415595 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:53.415575    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a09d35c4-d15f-45ad-855f-1b43f8fac1bf-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dljpp\" (UID: \"a09d35c4-d15f-45ad-855f-1b43f8fac1bf\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dljpp"
Jul 17 21:16:53.648880 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:53.648842    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dljpp"
Jul 17 21:16:53.787587 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:53.787562    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dljpp"]
Jul 17 21:16:53.788774 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:16:53.788749    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda09d35c4_d15f_45ad_855f_1b43f8fac1bf.slice/crio-9dc55a144c029fa086b48ebd9019a69d2f37d4d6dac7668468b1a9969bbcfa1a WatchSource:0}: Error finding container 9dc55a144c029fa086b48ebd9019a69d2f37d4d6dac7668468b1a9969bbcfa1a: Status 404 returned error can't find the container with id 9dc55a144c029fa086b48ebd9019a69d2f37d4d6dac7668468b1a9969bbcfa1a
Jul 17 21:16:53.889040 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:53.889006    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:53.993032 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:53.992935    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:16:54.158674 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:16:54.158630    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dljpp" event={"ID":"a09d35c4-d15f-45ad-855f-1b43f8fac1bf","Type":"ContainerStarted","Data":"9dc55a144c029fa086b48ebd9019a69d2f37d4d6dac7668468b1a9969bbcfa1a"}
Jul 17 21:17:01.144363 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:01.144268    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2m5tx"
Jul 17 21:17:01.144777 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:01.144473    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"
Jul 17 21:17:03.059468 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.059423    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"]
Jul 17 21:17:03.060104 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.059677    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh" podUID="94ee4c30-ebb4-4671-83bc-aa54e4546cdd" containerName="manager" containerID="cri-o://c3eb9cd332da8fd946c65fcaf01623938b840aac51a2761216a22b41320633dc" gracePeriod=2
Jul 17 21:17:03.061341 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:17:03.061303    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wspbh\" is forbidden: User \"system:node:ip-10-0-140-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-117.ec2.internal' and this object" podUID="94ee4c30-ebb4-4671-83bc-aa54e4546cdd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"
Jul 17 21:17:03.063694 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.063622    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"]
Jul 17 21:17:03.078117 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.078081    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"]
Jul 17 21:17:03.078383 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.078337    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp" podUID="72cac949-57ff-49b6-82b7-a4fe74884ac5" containerName="manager" containerID="cri-o://87e461cf8be4b47624b8b970ff640a9ff73c5153f65d9426b6152795ad3ac45b" gracePeriod=2
Jul 17 21:17:03.081127 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.081067    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"
Jul 17 21:17:03.093850 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.093783    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"]
Jul 17 21:17:03.096289 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:17:03.096234    2569 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"72cac949-57ff-49b6-82b7-a4fe74884ac5\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-4z9dp\": pods \"limitador-operator-controller-manager-859d7b78bd-4z9dp\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"
Jul 17 21:17:03.098792 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:17:03.098640    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wspbh\" is forbidden: User \"system:node:ip-10-0-140-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-117.ec2.internal' and this object" podUID="94ee4c30-ebb4-4671-83bc-aa54e4546cdd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"
Jul 17 21:17:03.100895 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:17:03.100856    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4z9dp\" is forbidden: User \"system:node:ip-10-0-140-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-117.ec2.internal' and this object" podUID="72cac949-57ff-49b6-82b7-a4fe74884ac5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"
Jul 17 21:17:03.101104 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.100901    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4gb9m"]
Jul 17 21:17:03.101458 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.101439    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72cac949-57ff-49b6-82b7-a4fe74884ac5" containerName="manager"
Jul 17 21:17:03.101751 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.101591    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="72cac949-57ff-49b6-82b7-a4fe74884ac5" containerName="manager"
Jul 17 21:17:03.101751 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.101638    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94ee4c30-ebb4-4671-83bc-aa54e4546cdd" containerName="manager"
Jul 17 21:17:03.101751 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.101647    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="94ee4c30-ebb4-4671-83bc-aa54e4546cdd" containerName="manager"
Jul 17 21:17:03.101942 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.101759    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="94ee4c30-ebb4-4671-83bc-aa54e4546cdd" containerName="manager"
Jul 17 21:17:03.101942 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.101776    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="72cac949-57ff-49b6-82b7-a4fe74884ac5" containerName="manager"
Jul 17 21:17:03.104302 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:17:03.104269    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wspbh\" is forbidden: User \"system:node:ip-10-0-140-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-117.ec2.internal' and this object" podUID="94ee4c30-ebb4-4671-83bc-aa54e4546cdd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"
Jul 17 21:17:03.105367 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.105345    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4gb9m"
Jul 17 21:17:03.107246 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:17:03.107216    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4z9dp\" is forbidden: User \"system:node:ip-10-0-140-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-117.ec2.internal' and this object" podUID="72cac949-57ff-49b6-82b7-a4fe74884ac5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"
Jul 17 21:17:03.108908 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:17:03.108876    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wspbh\" is forbidden: User \"system:node:ip-10-0-140-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-117.ec2.internal' and this object" podUID="94ee4c30-ebb4-4671-83bc-aa54e4546cdd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"
Jul 17 21:17:03.121152 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.121072    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4gb9m"]
Jul 17 21:17:03.134296 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.134210    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:03.204909 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.204873    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhccv\" (UniqueName: \"kubernetes.io/projected/e4124a45-e1e0-4c94-b4f0-bf65a28e54bd-kube-api-access-mhccv\") pod \"limitador-operator-controller-manager-859d7b78bd-4gb9m\" (UID: \"e4124a45-e1e0-4c94-b4f0-bf65a28e54bd\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4gb9m"
Jul 17 21:17:03.206219 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.206188    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:03.306024 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.305978    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhccv\" (UniqueName: \"kubernetes.io/projected/e4124a45-e1e0-4c94-b4f0-bf65a28e54bd-kube-api-access-mhccv\") pod \"limitador-operator-controller-manager-859d7b78bd-4gb9m\" (UID: \"e4124a45-e1e0-4c94-b4f0-bf65a28e54bd\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4gb9m"
Jul 17 21:17:03.318372 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.318294    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g"]
Jul 17 21:17:03.319586 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.319558    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhccv\" (UniqueName: \"kubernetes.io/projected/e4124a45-e1e0-4c94-b4f0-bf65a28e54bd-kube-api-access-mhccv\") pod \"limitador-operator-controller-manager-859d7b78bd-4gb9m\" (UID: \"e4124a45-e1e0-4c94-b4f0-bf65a28e54bd\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4gb9m"
Jul 17 21:17:03.323432 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.323396    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g"
Jul 17 21:17:03.326816 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:17:03.326778    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4z9dp\" is forbidden: User \"system:node:ip-10-0-140-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-117.ec2.internal' and this object" podUID="72cac949-57ff-49b6-82b7-a4fe74884ac5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"
Jul 17 21:17:03.329188 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:17:03.329159    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wspbh\" is forbidden: User \"system:node:ip-10-0-140-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-117.ec2.internal' and this object" podUID="94ee4c30-ebb4-4671-83bc-aa54e4546cdd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"
Jul 17 21:17:03.337627 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.337589    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g"]
Jul 17 21:17:03.407207 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.407165    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a39633e-78fe-434f-9263-b311f3a52c1b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vqh4g\" (UID: \"6a39633e-78fe-434f-9263-b311f3a52c1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g"
Jul 17 21:17:03.407384 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.407270    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhx4l\" (UniqueName: \"kubernetes.io/projected/6a39633e-78fe-434f-9263-b311f3a52c1b-kube-api-access-hhx4l\") pod \"kuadrant-operator-controller-manager-86878c769f-vqh4g\" (UID: \"6a39633e-78fe-434f-9263-b311f3a52c1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g"
Jul 17 21:17:03.475455 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.475413    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4gb9m"
Jul 17 21:17:03.508141 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.508095    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhx4l\" (UniqueName: \"kubernetes.io/projected/6a39633e-78fe-434f-9263-b311f3a52c1b-kube-api-access-hhx4l\") pod \"kuadrant-operator-controller-manager-86878c769f-vqh4g\" (UID: \"6a39633e-78fe-434f-9263-b311f3a52c1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g"
Jul 17 21:17:03.508317 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.508210    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a39633e-78fe-434f-9263-b311f3a52c1b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vqh4g\" (UID: \"6a39633e-78fe-434f-9263-b311f3a52c1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g"
Jul 17 21:17:03.508582 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.508566    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a39633e-78fe-434f-9263-b311f3a52c1b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vqh4g\" (UID: \"6a39633e-78fe-434f-9263-b311f3a52c1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g"
Jul 17 21:17:03.520909 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.520873    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhx4l\" (UniqueName: \"kubernetes.io/projected/6a39633e-78fe-434f-9263-b311f3a52c1b-kube-api-access-hhx4l\") pod \"kuadrant-operator-controller-manager-86878c769f-vqh4g\" (UID: \"6a39633e-78fe-434f-9263-b311f3a52c1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g"
Jul 17 21:17:03.641604 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:03.641563    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g"
Jul 17 21:17:06.085236 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:06.085195    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:06.187097 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:06.187054    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:08.703436 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:08.703382    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:08.804472 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:08.804425    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:12.289691 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:12.289637    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:12.364423 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:12.364345    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:15.082145 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:15.082108    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:15.182852 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:15.182811    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:16.317375 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:16.317334    2569 generic.go:356] "Generic (PLEG): container finished" podID="94ee4c30-ebb4-4671-83bc-aa54e4546cdd" containerID="c3eb9cd332da8fd946c65fcaf01623938b840aac51a2761216a22b41320633dc" exitCode=0
Jul 17 21:17:16.321562 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:16.321492    2569 generic.go:356] "Generic (PLEG): container finished" podID="72cac949-57ff-49b6-82b7-a4fe74884ac5" containerID="87e461cf8be4b47624b8b970ff640a9ff73c5153f65d9426b6152795ad3ac45b" exitCode=0
Jul 17 21:17:16.479701 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:16.479669    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g"]
Jul 17 21:17:16.481584 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:16.481563    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"
Jul 17 21:17:16.483237 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:17:16.483185    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4z9dp\" is forbidden: User \"system:node:ip-10-0-140-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-117.ec2.internal' and this object" podUID="72cac949-57ff-49b6-82b7-a4fe74884ac5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"
Jul 17 21:17:16.484027 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:17:16.483985    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a39633e_78fe_434f_9263_b311f3a52c1b.slice/crio-60f3cbf3623732cc3fcbab8c1ecb9eb86e04678365eb5685c7a91a42fae7595e WatchSource:0}: Error finding container 60f3cbf3623732cc3fcbab8c1ecb9eb86e04678365eb5685c7a91a42fae7595e: Status 404 returned error can't find the container with id 60f3cbf3623732cc3fcbab8c1ecb9eb86e04678365eb5685c7a91a42fae7595e
Jul 17 21:17:16.487460 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:16.487380    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"
Jul 17 21:17:16.489194 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:17:16.489161    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4z9dp\" is forbidden: User \"system:node:ip-10-0-140-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-117.ec2.internal' and this object" podUID="72cac949-57ff-49b6-82b7-a4fe74884ac5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"
Jul 17 21:17:16.490790 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:17:16.490765    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wspbh\" is forbidden: User \"system:node:ip-10-0-140-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-117.ec2.internal' and this object" podUID="94ee4c30-ebb4-4671-83bc-aa54e4546cdd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"
Jul 17 21:17:16.503052 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:16.503029    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4gb9m"]
Jul 17 21:17:16.504807 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:17:16.504775    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4124a45_e1e0_4c94_b4f0_bf65a28e54bd.slice/crio-e964f621b25403a197ffc106e46358efee32a66b737d214e80ca4bc79132cf5b WatchSource:0}: Error finding container e964f621b25403a197ffc106e46358efee32a66b737d214e80ca4bc79132cf5b: Status 404 returned error can't find the container with id e964f621b25403a197ffc106e46358efee32a66b737d214e80ca4bc79132cf5b
Jul 17 21:17:16.634688 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:16.634637    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckwq8\" (UniqueName: \"kubernetes.io/projected/94ee4c30-ebb4-4671-83bc-aa54e4546cdd-kube-api-access-ckwq8\") pod \"94ee4c30-ebb4-4671-83bc-aa54e4546cdd\" (UID: \"94ee4c30-ebb4-4671-83bc-aa54e4546cdd\") "
Jul 17 21:17:16.634688 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:16.634699    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94ee4c30-ebb4-4671-83bc-aa54e4546cdd-extensions-socket-volume\") pod \"94ee4c30-ebb4-4671-83bc-aa54e4546cdd\" (UID: \"94ee4c30-ebb4-4671-83bc-aa54e4546cdd\") "
Jul 17 21:17:16.634914 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:16.634718    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf6j4\" (UniqueName: \"kubernetes.io/projected/72cac949-57ff-49b6-82b7-a4fe74884ac5-kube-api-access-kf6j4\") pod \"72cac949-57ff-49b6-82b7-a4fe74884ac5\" (UID: \"72cac949-57ff-49b6-82b7-a4fe74884ac5\") "
Jul 17 21:17:16.635286 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:16.635253    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ee4c30-ebb4-4671-83bc-aa54e4546cdd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "94ee4c30-ebb4-4671-83bc-aa54e4546cdd" (UID: "94ee4c30-ebb4-4671-83bc-aa54e4546cdd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:17:16.636812 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:16.636791    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ee4c30-ebb4-4671-83bc-aa54e4546cdd-kube-api-access-ckwq8" (OuterVolumeSpecName: "kube-api-access-ckwq8") pod "94ee4c30-ebb4-4671-83bc-aa54e4546cdd" (UID: "94ee4c30-ebb4-4671-83bc-aa54e4546cdd"). InnerVolumeSpecName "kube-api-access-ckwq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:17:16.636885 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:16.636816    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72cac949-57ff-49b6-82b7-a4fe74884ac5-kube-api-access-kf6j4" (OuterVolumeSpecName: "kube-api-access-kf6j4") pod "72cac949-57ff-49b6-82b7-a4fe74884ac5" (UID: "72cac949-57ff-49b6-82b7-a4fe74884ac5"). InnerVolumeSpecName "kube-api-access-kf6j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:17:16.654819 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:17:16.654782    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wspbh\" is forbidden: User \"system:node:ip-10-0-140-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-117.ec2.internal' and this object" podUID="94ee4c30-ebb4-4671-83bc-aa54e4546cdd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"
Jul 17 21:17:16.656209 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:17:16.656184    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4z9dp\" is forbidden: User \"system:node:ip-10-0-140-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-117.ec2.internal' and this object" podUID="72cac949-57ff-49b6-82b7-a4fe74884ac5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"
Jul 17 21:17:16.657007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:16.656981    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72cac949-57ff-49b6-82b7-a4fe74884ac5" path="/var/lib/kubelet/pods/72cac949-57ff-49b6-82b7-a4fe74884ac5/volumes"
Jul 17 21:17:16.657411 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:16.657380    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94ee4c30-ebb4-4671-83bc-aa54e4546cdd" path="/var/lib/kubelet/pods/94ee4c30-ebb4-4671-83bc-aa54e4546cdd/volumes"
Jul 17 21:17:16.735762 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:16.735729    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckwq8\" (UniqueName: \"kubernetes.io/projected/94ee4c30-ebb4-4671-83bc-aa54e4546cdd-kube-api-access-ckwq8\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:17:16.735762 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:16.735759    2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94ee4c30-ebb4-4671-83bc-aa54e4546cdd-extensions-socket-volume\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:17:16.735762 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:16.735772    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf6j4\" (UniqueName: \"kubernetes.io/projected/72cac949-57ff-49b6-82b7-a4fe74884ac5-kube-api-access-kf6j4\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:17:17.328622 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:17.328581    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dljpp" event={"ID":"a09d35c4-d15f-45ad-855f-1b43f8fac1bf","Type":"ContainerStarted","Data":"822501b3f2eab6c95c8d17908ea509d9392394defb84a66a656eaa5cc56822e8"}
Jul 17 21:17:17.329862 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:17.329841    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"
Jul 17 21:17:17.329991 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:17.329873    2569 scope.go:117] "RemoveContainer" containerID="87e461cf8be4b47624b8b970ff640a9ff73c5153f65d9426b6152795ad3ac45b"
Jul 17 21:17:17.331889 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:17.331863    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g" event={"ID":"6a39633e-78fe-434f-9263-b311f3a52c1b","Type":"ContainerStarted","Data":"48d380e2bfb60ff8a21ffab15037d7f5919160c86f846d015d7c047359ea5768"}
Jul 17 21:17:17.332070 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:17.332053    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g" event={"ID":"6a39633e-78fe-434f-9263-b311f3a52c1b","Type":"ContainerStarted","Data":"60f3cbf3623732cc3fcbab8c1ecb9eb86e04678365eb5685c7a91a42fae7595e"}
Jul 17 21:17:17.332203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:17.332175    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g"
Jul 17 21:17:17.333349 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:17.333332    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"
Jul 17 21:17:17.334890 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:17.334865    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4gb9m" event={"ID":"e4124a45-e1e0-4c94-b4f0-bf65a28e54bd","Type":"ContainerStarted","Data":"8463c0a1520f2df6cf00a4d7345f603b242eca48d060b5df6f0cd4c013d39586"}
Jul 17 21:17:17.334990 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:17.334913    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4gb9m" event={"ID":"e4124a45-e1e0-4c94-b4f0-bf65a28e54bd","Type":"ContainerStarted","Data":"e964f621b25403a197ffc106e46358efee32a66b737d214e80ca4bc79132cf5b"}
Jul 17 21:17:17.334990 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:17.334943    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4gb9m"
Jul 17 21:17:17.340058 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:17.340042    2569 scope.go:117] "RemoveContainer" containerID="c3eb9cd332da8fd946c65fcaf01623938b840aac51a2761216a22b41320633dc"
Jul 17 21:17:17.345249 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:17.345210    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dljpp" podStartSLOduration=2.730976358 podStartE2EDuration="25.345198126s" totalImagesPullingTime="22.614221768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:17.344279795 +0000 UTC m=+551.103846491" watchObservedRunningTime="2026-07-17 21:17:17.345198126 +0000 UTC m=+551.104764819"
Jul 17 21:17:17.346068 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:17:17.346043    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4z9dp\" is forbidden: User \"system:node:ip-10-0-140-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-117.ec2.internal' and this object" podUID="72cac949-57ff-49b6-82b7-a4fe74884ac5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4z9dp"
Jul 17 21:17:17.347534 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:17:17.347512    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wspbh\" is forbidden: User \"system:node:ip-10-0-140-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-117.ec2.internal' and this object" podUID="94ee4c30-ebb4-4671-83bc-aa54e4546cdd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wspbh"
Jul 17 21:17:17.369464 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:17.369390    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g" podStartSLOduration=14.369375381 podStartE2EDuration="14.369375381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:17.366837843 +0000 UTC m=+551.126404542" watchObservedRunningTime="2026-07-17 21:17:17.369375381 +0000 UTC m=+551.128942076"
Jul 17 21:17:17.385354 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:17.385300    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4gb9m" podStartSLOduration=14.385285376 podStartE2EDuration="14.385285376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:17.384136676 +0000 UTC m=+551.143703369" watchObservedRunningTime="2026-07-17 21:17:17.385285376 +0000 UTC m=+551.144852069"
Jul 17 21:17:25.284679 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:25.284645    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:25.339205 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:25.339165    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:25.683048 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:25.683009    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:25.738754 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:25.738717    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:28.348547 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:28.348513    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4gb9m"
Jul 17 21:17:28.951101 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:28.951058    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:29.043081 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:29.043042    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:29.355684 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:29.355652    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g"
Jul 17 21:17:45.775757 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:45.775692    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:46.157207 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:46.157168    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:53.449601 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:53.449560    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:53.549466 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:53.549410    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:55.756454 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:55.756390    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:55.879467 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:55.879429    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:55.930147 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:55.930111    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:56.021080 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.020993    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:17:56.064034 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.063970    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:17:56.064179 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.064056    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:17:56.064179 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.064074    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jb626"
Jul 17 21:17:56.066068 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.066043    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 17 21:17:56.174874 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.174839    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b6bc014-05c4-4303-88c4-8536484d2681-config-file\") pod \"limitador-limitador-54d6d988d5-jb626\" (UID: \"6b6bc014-05c4-4303-88c4-8536484d2681\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jb626"
Jul 17 21:17:56.175040 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.174903    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c25rp\" (UniqueName: \"kubernetes.io/projected/6b6bc014-05c4-4303-88c4-8536484d2681-kube-api-access-c25rp\") pod \"limitador-limitador-54d6d988d5-jb626\" (UID: \"6b6bc014-05c4-4303-88c4-8536484d2681\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jb626"
Jul 17 21:17:56.276216 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.276131    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b6bc014-05c4-4303-88c4-8536484d2681-config-file\") pod \"limitador-limitador-54d6d988d5-jb626\" (UID: \"6b6bc014-05c4-4303-88c4-8536484d2681\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jb626"
Jul 17 21:17:56.276216 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.276198    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c25rp\" (UniqueName: \"kubernetes.io/projected/6b6bc014-05c4-4303-88c4-8536484d2681-kube-api-access-c25rp\") pod \"limitador-limitador-54d6d988d5-jb626\" (UID: \"6b6bc014-05c4-4303-88c4-8536484d2681\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jb626"
Jul 17 21:17:56.276803 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.276784    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b6bc014-05c4-4303-88c4-8536484d2681-config-file\") pod \"limitador-limitador-54d6d988d5-jb626\" (UID: \"6b6bc014-05c4-4303-88c4-8536484d2681\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jb626"
Jul 17 21:17:56.284320 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.284296    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c25rp\" (UniqueName: \"kubernetes.io/projected/6b6bc014-05c4-4303-88c4-8536484d2681-kube-api-access-c25rp\") pod \"limitador-limitador-54d6d988d5-jb626\" (UID: \"6b6bc014-05c4-4303-88c4-8536484d2681\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jb626"
Jul 17 21:17:56.375182 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.375146    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jb626"
Jul 17 21:17:56.507877 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.507853    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:17:56.509236 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:17:56.509214    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6bc014_05c4_4303_88c4_8536484d2681.slice/crio-48fdf42c25c9dda96462fad4d0c93f135890f130086e028c0dfeed3f01e87db2 WatchSource:0}: Error finding container 48fdf42c25c9dda96462fad4d0c93f135890f130086e028c0dfeed3f01e87db2: Status 404 returned error can't find the container with id 48fdf42c25c9dda96462fad4d0c93f135890f130086e028c0dfeed3f01e87db2
Jul 17 21:17:56.564917 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.564839    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jb626" event={"ID":"6b6bc014-05c4-4303-88c4-8536484d2681","Type":"ContainerStarted","Data":"48fdf42c25c9dda96462fad4d0c93f135890f130086e028c0dfeed3f01e87db2"}
Jul 17 21:17:56.573181 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.573142    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-w6k4z"]
Jul 17 21:17:56.578028 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.578009    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-w6k4z"
Jul 17 21:17:56.580162 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.580140    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nrgzg\""
Jul 17 21:17:56.582266 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.582240    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-w6k4z"]
Jul 17 21:17:56.608386 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.608354    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:56.679727 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.679697    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s48ds\" (UniqueName: \"kubernetes.io/projected/e9a58edc-d421-4404-9ab0-4f7dd906b29a-kube-api-access-s48ds\") pod \"authorino-58cd955675-w6k4z\" (UID: \"e9a58edc-d421-4404-9ab0-4f7dd906b29a\") " pod="kuadrant-system/authorino-58cd955675-w6k4z"
Jul 17 21:17:56.708129 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.708091    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:17:56.780128 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.780098    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s48ds\" (UniqueName: \"kubernetes.io/projected/e9a58edc-d421-4404-9ab0-4f7dd906b29a-kube-api-access-s48ds\") pod \"authorino-58cd955675-w6k4z\" (UID: \"e9a58edc-d421-4404-9ab0-4f7dd906b29a\") " pod="kuadrant-system/authorino-58cd955675-w6k4z"
Jul 17 21:17:56.789879 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.789856    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s48ds\" (UniqueName: \"kubernetes.io/projected/e9a58edc-d421-4404-9ab0-4f7dd906b29a-kube-api-access-s48ds\") pod \"authorino-58cd955675-w6k4z\" (UID: \"e9a58edc-d421-4404-9ab0-4f7dd906b29a\") " pod="kuadrant-system/authorino-58cd955675-w6k4z"
Jul 17 21:17:56.889544 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:56.889506    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-w6k4z"
Jul 17 21:17:57.069085 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:57.069055    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-w6k4z"]
Jul 17 21:17:57.071607 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:17:57.071562    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9a58edc_d421_4404_9ab0_4f7dd906b29a.slice/crio-62a2dfc2df70c33c4f52374ffbd4a1452d8e6fa9b6f5df163a203776ef6105fe WatchSource:0}: Error finding container 62a2dfc2df70c33c4f52374ffbd4a1452d8e6fa9b6f5df163a203776ef6105fe: Status 404 returned error can't find the container with id 62a2dfc2df70c33c4f52374ffbd4a1452d8e6fa9b6f5df163a203776ef6105fe
Jul 17 21:17:57.572414 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:57.572372    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-w6k4z" event={"ID":"e9a58edc-d421-4404-9ab0-4f7dd906b29a","Type":"ContainerStarted","Data":"62a2dfc2df70c33c4f52374ffbd4a1452d8e6fa9b6f5df163a203776ef6105fe"}
Jul 17 21:17:59.591534 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:59.591479    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jb626" event={"ID":"6b6bc014-05c4-4303-88c4-8536484d2681","Type":"ContainerStarted","Data":"f6cb468c4e25909dbc2c4955aa1194b1a7203e16b29a3f22ad752e4c2733eb87"}
Jul 17 21:17:59.592061 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:59.591792    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jb626"
Jul 17 21:17:59.611693 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:17:59.611632    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-jb626" podStartSLOduration=1.125156173 podStartE2EDuration="3.611611732s" totalImagesPullingTime="2.486455559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:59.608692729 +0000 UTC m=+593.368259424" watchObservedRunningTime="2026-07-17 21:17:59.611611732 +0000 UTC m=+593.371178429"
Jul 17 21:18:01.604771 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:01.604725    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-w6k4z" event={"ID":"e9a58edc-d421-4404-9ab0-4f7dd906b29a","Type":"ContainerStarted","Data":"99fb75b71ef158956e78b7a8973e8ab68042c73076798d4a55321f78d5c624ff"}
Jul 17 21:18:01.620184 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:01.620140    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-w6k4z" podStartSLOduration=1.37787458 podStartE2EDuration="5.620126586s" totalImagesPullingTime="4.242252006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:18:01.6178418 +0000 UTC m=+595.377408494" watchObservedRunningTime="2026-07-17 21:18:01.620126586 +0000 UTC m=+595.379693309"
Jul 17 21:18:01.858824 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:01.858721    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-w6k4z"]
Jul 17 21:18:03.615741 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:03.615680    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-w6k4z" podUID="e9a58edc-d421-4404-9ab0-4f7dd906b29a" containerName="authorino" containerID="cri-o://99fb75b71ef158956e78b7a8973e8ab68042c73076798d4a55321f78d5c624ff" gracePeriod=30
Jul 17 21:18:03.861960 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:03.861931    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-w6k4z"
Jul 17 21:18:03.951327 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:03.951215    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s48ds\" (UniqueName: \"kubernetes.io/projected/e9a58edc-d421-4404-9ab0-4f7dd906b29a-kube-api-access-s48ds\") pod \"e9a58edc-d421-4404-9ab0-4f7dd906b29a\" (UID: \"e9a58edc-d421-4404-9ab0-4f7dd906b29a\") "
Jul 17 21:18:03.953304 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:03.953277    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a58edc-d421-4404-9ab0-4f7dd906b29a-kube-api-access-s48ds" (OuterVolumeSpecName: "kube-api-access-s48ds") pod "e9a58edc-d421-4404-9ab0-4f7dd906b29a" (UID: "e9a58edc-d421-4404-9ab0-4f7dd906b29a"). InnerVolumeSpecName "kube-api-access-s48ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:18:04.052368 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:04.052326    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s48ds\" (UniqueName: \"kubernetes.io/projected/e9a58edc-d421-4404-9ab0-4f7dd906b29a-kube-api-access-s48ds\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:18:04.622861 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:04.622824    2569 generic.go:356] "Generic (PLEG): container finished" podID="e9a58edc-d421-4404-9ab0-4f7dd906b29a" containerID="99fb75b71ef158956e78b7a8973e8ab68042c73076798d4a55321f78d5c624ff" exitCode=0
Jul 17 21:18:04.623305 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:04.622872    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-w6k4z"
Jul 17 21:18:04.623305 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:04.622903    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-w6k4z" event={"ID":"e9a58edc-d421-4404-9ab0-4f7dd906b29a","Type":"ContainerDied","Data":"99fb75b71ef158956e78b7a8973e8ab68042c73076798d4a55321f78d5c624ff"}
Jul 17 21:18:04.623305 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:04.622951    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-w6k4z" event={"ID":"e9a58edc-d421-4404-9ab0-4f7dd906b29a","Type":"ContainerDied","Data":"62a2dfc2df70c33c4f52374ffbd4a1452d8e6fa9b6f5df163a203776ef6105fe"}
Jul 17 21:18:04.623305 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:04.622974    2569 scope.go:117] "RemoveContainer" containerID="99fb75b71ef158956e78b7a8973e8ab68042c73076798d4a55321f78d5c624ff"
Jul 17 21:18:04.635271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:04.635256    2569 scope.go:117] "RemoveContainer" containerID="99fb75b71ef158956e78b7a8973e8ab68042c73076798d4a55321f78d5c624ff"
Jul 17 21:18:04.635631 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:18:04.635608    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99fb75b71ef158956e78b7a8973e8ab68042c73076798d4a55321f78d5c624ff\": container with ID starting with 99fb75b71ef158956e78b7a8973e8ab68042c73076798d4a55321f78d5c624ff not found: ID does not exist" containerID="99fb75b71ef158956e78b7a8973e8ab68042c73076798d4a55321f78d5c624ff"
Jul 17 21:18:04.635702 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:04.635642    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99fb75b71ef158956e78b7a8973e8ab68042c73076798d4a55321f78d5c624ff"} err="failed to get container status \"99fb75b71ef158956e78b7a8973e8ab68042c73076798d4a55321f78d5c624ff\": rpc error: code = NotFound desc = could not find container \"99fb75b71ef158956e78b7a8973e8ab68042c73076798d4a55321f78d5c624ff\": container with ID starting with 99fb75b71ef158956e78b7a8973e8ab68042c73076798d4a55321f78d5c624ff not found: ID does not exist"
Jul 17 21:18:04.645362 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:04.645338    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-w6k4z"]
Jul 17 21:18:04.656541 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:04.656520    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-w6k4z"]
Jul 17 21:18:06.600742 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:06.600712    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:18:06.602115 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:06.602094    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:18:06.656357 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:06.656327    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9a58edc-d421-4404-9ab0-4f7dd906b29a" path="/var/lib/kubelet/pods/e9a58edc-d421-4404-9ab0-4f7dd906b29a/volumes"
Jul 17 21:18:10.600459 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:10.600392    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jb626"
Jul 17 21:18:11.869968 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:11.869928    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:18:13.288314 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:13.288283    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:18:26.284984 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.284945    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:18:26.486711 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.486678    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2pn7m"]
Jul 17 21:18:26.487092 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.487079    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9a58edc-d421-4404-9ab0-4f7dd906b29a" containerName="authorino"
Jul 17 21:18:26.487135 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.487093    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9a58edc-d421-4404-9ab0-4f7dd906b29a" containerName="authorino"
Jul 17 21:18:26.487170 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.487161    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9a58edc-d421-4404-9ab0-4f7dd906b29a" containerName="authorino"
Jul 17 21:18:26.491828 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.491802    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2pn7m"
Jul 17 21:18:26.494002 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.493975    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nrgzg\""
Jul 17 21:18:26.497204 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.497180    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2pn7m"]
Jul 17 21:18:26.543327 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.543240    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6dpc\" (UniqueName: \"kubernetes.io/projected/50d69fdc-f89d-4a21-87c4-95275d5bae99-kube-api-access-l6dpc\") pod \"authorino-5d4b67c446-2pn7m\" (UID: \"50d69fdc-f89d-4a21-87c4-95275d5bae99\") " pod="kuadrant-system/authorino-5d4b67c446-2pn7m"
Jul 17 21:18:26.644430 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.644385    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6dpc\" (UniqueName: \"kubernetes.io/projected/50d69fdc-f89d-4a21-87c4-95275d5bae99-kube-api-access-l6dpc\") pod \"authorino-5d4b67c446-2pn7m\" (UID: \"50d69fdc-f89d-4a21-87c4-95275d5bae99\") " pod="kuadrant-system/authorino-5d4b67c446-2pn7m"
Jul 17 21:18:26.652205 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.652173    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6dpc\" (UniqueName: \"kubernetes.io/projected/50d69fdc-f89d-4a21-87c4-95275d5bae99-kube-api-access-l6dpc\") pod \"authorino-5d4b67c446-2pn7m\" (UID: \"50d69fdc-f89d-4a21-87c4-95275d5bae99\") " pod="kuadrant-system/authorino-5d4b67c446-2pn7m"
Jul 17 21:18:26.717013 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.716976    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2pn7m"]
Jul 17 21:18:26.717248 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.717236    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2pn7m"
Jul 17 21:18:26.741162 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.741130    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d4f9f9554-nldt8"]
Jul 17 21:18:26.746258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.746232    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d4f9f9554-nldt8"
Jul 17 21:18:26.748563 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.748536    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d4f9f9554-nldt8"]
Jul 17 21:18:26.813144 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.813063    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:18:26.846213 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.846174    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbj87\" (UniqueName: \"kubernetes.io/projected/3151c803-ba20-44aa-ab7e-5a301b26e61a-kube-api-access-dbj87\") pod \"authorino-d4f9f9554-nldt8\" (UID: \"3151c803-ba20-44aa-ab7e-5a301b26e61a\") " pod="kuadrant-system/authorino-d4f9f9554-nldt8"
Jul 17 21:18:26.868516 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.868450    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2pn7m"]
Jul 17 21:18:26.871394 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:18:26.871365    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50d69fdc_f89d_4a21_87c4_95275d5bae99.slice/crio-bde820936ab2c47ac2eb4090efc7cb674708f9ca0af09ed64fc2b17ff6bf262a WatchSource:0}: Error finding container bde820936ab2c47ac2eb4090efc7cb674708f9ca0af09ed64fc2b17ff6bf262a: Status 404 returned error can't find the container with id bde820936ab2c47ac2eb4090efc7cb674708f9ca0af09ed64fc2b17ff6bf262a
Jul 17 21:18:26.872753 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.872733    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 21:18:26.947497 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.947456    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbj87\" (UniqueName: \"kubernetes.io/projected/3151c803-ba20-44aa-ab7e-5a301b26e61a-kube-api-access-dbj87\") pod \"authorino-d4f9f9554-nldt8\" (UID: \"3151c803-ba20-44aa-ab7e-5a301b26e61a\") " pod="kuadrant-system/authorino-d4f9f9554-nldt8"
Jul 17 21:18:26.955068 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:26.955041    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbj87\" (UniqueName: \"kubernetes.io/projected/3151c803-ba20-44aa-ab7e-5a301b26e61a-kube-api-access-dbj87\") pod \"authorino-d4f9f9554-nldt8\" (UID: \"3151c803-ba20-44aa-ab7e-5a301b26e61a\") " pod="kuadrant-system/authorino-d4f9f9554-nldt8"
Jul 17 21:18:27.058732 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.058690    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d4f9f9554-nldt8"
Jul 17 21:18:27.158360 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.158322    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d4f9f9554-nldt8"]
Jul 17 21:18:27.186191 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.186162    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-677f9477cd-7tt7f"]
Jul 17 21:18:27.190647 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.190629    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-677f9477cd-7tt7f"
Jul 17 21:18:27.192502 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.192478    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 17 21:18:27.196514 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.196477    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-677f9477cd-7tt7f"]
Jul 17 21:18:27.200004 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.199169    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d4f9f9554-nldt8"]
Jul 17 21:18:27.202750 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:18:27.202724    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3151c803_ba20_44aa_ab7e_5a301b26e61a.slice/crio-dcd8ec112238bf04e4cd16c37f24f77055ae6bd92869d13874e65b927198f187 WatchSource:0}: Error finding container dcd8ec112238bf04e4cd16c37f24f77055ae6bd92869d13874e65b927198f187: Status 404 returned error can't find the container with id dcd8ec112238bf04e4cd16c37f24f77055ae6bd92869d13874e65b927198f187
Jul 17 21:18:27.251364 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.251328    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2cqh\" (UniqueName: \"kubernetes.io/projected/7e338ed9-bdc1-41cc-b12b-998cc168011e-kube-api-access-s2cqh\") pod \"authorino-677f9477cd-7tt7f\" (UID: \"7e338ed9-bdc1-41cc-b12b-998cc168011e\") " pod="kuadrant-system/authorino-677f9477cd-7tt7f"
Jul 17 21:18:27.251545 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.251380    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e338ed9-bdc1-41cc-b12b-998cc168011e-tls-cert\") pod \"authorino-677f9477cd-7tt7f\" (UID: \"7e338ed9-bdc1-41cc-b12b-998cc168011e\") " pod="kuadrant-system/authorino-677f9477cd-7tt7f"
Jul 17 21:18:27.353075 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.353033    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e338ed9-bdc1-41cc-b12b-998cc168011e-tls-cert\") pod \"authorino-677f9477cd-7tt7f\" (UID: \"7e338ed9-bdc1-41cc-b12b-998cc168011e\") " pod="kuadrant-system/authorino-677f9477cd-7tt7f"
Jul 17 21:18:27.353611 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.353152    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2cqh\" (UniqueName: \"kubernetes.io/projected/7e338ed9-bdc1-41cc-b12b-998cc168011e-kube-api-access-s2cqh\") pod \"authorino-677f9477cd-7tt7f\" (UID: \"7e338ed9-bdc1-41cc-b12b-998cc168011e\") " pod="kuadrant-system/authorino-677f9477cd-7tt7f"
Jul 17 21:18:27.355873 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.355850    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e338ed9-bdc1-41cc-b12b-998cc168011e-tls-cert\") pod \"authorino-677f9477cd-7tt7f\" (UID: \"7e338ed9-bdc1-41cc-b12b-998cc168011e\") " pod="kuadrant-system/authorino-677f9477cd-7tt7f"
Jul 17 21:18:27.364337 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.364301    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2cqh\" (UniqueName: \"kubernetes.io/projected/7e338ed9-bdc1-41cc-b12b-998cc168011e-kube-api-access-s2cqh\") pod \"authorino-677f9477cd-7tt7f\" (UID: \"7e338ed9-bdc1-41cc-b12b-998cc168011e\") " pod="kuadrant-system/authorino-677f9477cd-7tt7f"
Jul 17 21:18:27.503498 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.503453    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-677f9477cd-7tt7f"
Jul 17 21:18:27.663710 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.663675    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-677f9477cd-7tt7f"]
Jul 17 21:18:27.666242 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:18:27.666210    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e338ed9_bdc1_41cc_b12b_998cc168011e.slice/crio-fe15bb63cfbaa16efedd7228dc05e97d68a37558913d37898cd2df24d7796c76 WatchSource:0}: Error finding container fe15bb63cfbaa16efedd7228dc05e97d68a37558913d37898cd2df24d7796c76: Status 404 returned error can't find the container with id fe15bb63cfbaa16efedd7228dc05e97d68a37558913d37898cd2df24d7796c76
Jul 17 21:18:27.748682 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.748636    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2pn7m" event={"ID":"50d69fdc-f89d-4a21-87c4-95275d5bae99","Type":"ContainerStarted","Data":"5da5ada75a83df885cb17218a663991080abf91f5184b98c7bafc1642f50e699"}
Jul 17 21:18:27.748873 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.748693    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2pn7m" event={"ID":"50d69fdc-f89d-4a21-87c4-95275d5bae99","Type":"ContainerStarted","Data":"bde820936ab2c47ac2eb4090efc7cb674708f9ca0af09ed64fc2b17ff6bf262a"}
Jul 17 21:18:27.748873 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.748707    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-2pn7m" podUID="50d69fdc-f89d-4a21-87c4-95275d5bae99" containerName="authorino" containerID="cri-o://5da5ada75a83df885cb17218a663991080abf91f5184b98c7bafc1642f50e699" gracePeriod=30
Jul 17 21:18:27.750299 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.750273    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-d4f9f9554-nldt8" podUID="3151c803-ba20-44aa-ab7e-5a301b26e61a" containerName="authorino" containerID="cri-o://c84d8603ae3a8e9cdff9e9e5f71a3c661b1a5678137e2d58e48a47bb335c34ba" gracePeriod=30
Jul 17 21:18:27.750453 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.750280    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d4f9f9554-nldt8" event={"ID":"3151c803-ba20-44aa-ab7e-5a301b26e61a","Type":"ContainerStarted","Data":"c84d8603ae3a8e9cdff9e9e5f71a3c661b1a5678137e2d58e48a47bb335c34ba"}
Jul 17 21:18:27.750453 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.750417    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d4f9f9554-nldt8" event={"ID":"3151c803-ba20-44aa-ab7e-5a301b26e61a","Type":"ContainerStarted","Data":"dcd8ec112238bf04e4cd16c37f24f77055ae6bd92869d13874e65b927198f187"}
Jul 17 21:18:27.752105 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.751691    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-677f9477cd-7tt7f" event={"ID":"7e338ed9-bdc1-41cc-b12b-998cc168011e","Type":"ContainerStarted","Data":"fe15bb63cfbaa16efedd7228dc05e97d68a37558913d37898cd2df24d7796c76"}
Jul 17 21:18:27.763434 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.763200    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-2pn7m" podStartSLOduration=1.416515521 podStartE2EDuration="1.763180756s" totalImagesPullingTime="346.665235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:18:27.762367352 +0000 UTC m=+621.521934046" watchObservedRunningTime="2026-07-17 21:18:27.763180756 +0000 UTC m=+621.522747450"
Jul 17 21:18:27.777473 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:27.777429    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-d4f9f9554-nldt8" podStartSLOduration=1.464970061 podStartE2EDuration="1.777412711s" totalImagesPullingTime="312.44265ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:18:27.774982583 +0000 UTC m=+621.534549276" watchObservedRunningTime="2026-07-17 21:18:27.777412711 +0000 UTC m=+621.536979399"
Jul 17 21:18:28.025854 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.025824    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2pn7m"
Jul 17 21:18:28.028920 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.028899    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d4f9f9554-nldt8"
Jul 17 21:18:28.059355 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.059325    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6dpc\" (UniqueName: \"kubernetes.io/projected/50d69fdc-f89d-4a21-87c4-95275d5bae99-kube-api-access-l6dpc\") pod \"50d69fdc-f89d-4a21-87c4-95275d5bae99\" (UID: \"50d69fdc-f89d-4a21-87c4-95275d5bae99\") "
Jul 17 21:18:28.059355 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.059358    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbj87\" (UniqueName: \"kubernetes.io/projected/3151c803-ba20-44aa-ab7e-5a301b26e61a-kube-api-access-dbj87\") pod \"3151c803-ba20-44aa-ab7e-5a301b26e61a\" (UID: \"3151c803-ba20-44aa-ab7e-5a301b26e61a\") "
Jul 17 21:18:28.061691 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.061651    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d69fdc-f89d-4a21-87c4-95275d5bae99-kube-api-access-l6dpc" (OuterVolumeSpecName: "kube-api-access-l6dpc") pod "50d69fdc-f89d-4a21-87c4-95275d5bae99" (UID: "50d69fdc-f89d-4a21-87c4-95275d5bae99"). InnerVolumeSpecName "kube-api-access-l6dpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:18:28.061813 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.061702    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3151c803-ba20-44aa-ab7e-5a301b26e61a-kube-api-access-dbj87" (OuterVolumeSpecName: "kube-api-access-dbj87") pod "3151c803-ba20-44aa-ab7e-5a301b26e61a" (UID: "3151c803-ba20-44aa-ab7e-5a301b26e61a"). InnerVolumeSpecName "kube-api-access-dbj87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:18:28.161132 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.161083    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6dpc\" (UniqueName: \"kubernetes.io/projected/50d69fdc-f89d-4a21-87c4-95275d5bae99-kube-api-access-l6dpc\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:18:28.161132 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.161137    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbj87\" (UniqueName: \"kubernetes.io/projected/3151c803-ba20-44aa-ab7e-5a301b26e61a-kube-api-access-dbj87\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:18:28.758516 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.758472    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-677f9477cd-7tt7f" event={"ID":"7e338ed9-bdc1-41cc-b12b-998cc168011e","Type":"ContainerStarted","Data":"ce0050cd0694a79df8b8e766502a7cd59e22bc773daf6a9adb20a59c4fb217ab"}
Jul 17 21:18:28.759782 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.759758    2569 generic.go:356] "Generic (PLEG): container finished" podID="50d69fdc-f89d-4a21-87c4-95275d5bae99" containerID="5da5ada75a83df885cb17218a663991080abf91f5184b98c7bafc1642f50e699" exitCode=0
Jul 17 21:18:28.759915 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.759790    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2pn7m"
Jul 17 21:18:28.759915 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.759833    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2pn7m" event={"ID":"50d69fdc-f89d-4a21-87c4-95275d5bae99","Type":"ContainerDied","Data":"5da5ada75a83df885cb17218a663991080abf91f5184b98c7bafc1642f50e699"}
Jul 17 21:18:28.759915 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.759867    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2pn7m" event={"ID":"50d69fdc-f89d-4a21-87c4-95275d5bae99","Type":"ContainerDied","Data":"bde820936ab2c47ac2eb4090efc7cb674708f9ca0af09ed64fc2b17ff6bf262a"}
Jul 17 21:18:28.759915 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.759890    2569 scope.go:117] "RemoveContainer" containerID="5da5ada75a83df885cb17218a663991080abf91f5184b98c7bafc1642f50e699"
Jul 17 21:18:28.761210 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.761190    2569 generic.go:356] "Generic (PLEG): container finished" podID="3151c803-ba20-44aa-ab7e-5a301b26e61a" containerID="c84d8603ae3a8e9cdff9e9e5f71a3c661b1a5678137e2d58e48a47bb335c34ba" exitCode=0
Jul 17 21:18:28.761299 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.761220    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d4f9f9554-nldt8"
Jul 17 21:18:28.761299 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.761234    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d4f9f9554-nldt8" event={"ID":"3151c803-ba20-44aa-ab7e-5a301b26e61a","Type":"ContainerDied","Data":"c84d8603ae3a8e9cdff9e9e5f71a3c661b1a5678137e2d58e48a47bb335c34ba"}
Jul 17 21:18:28.761299 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.761259    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d4f9f9554-nldt8" event={"ID":"3151c803-ba20-44aa-ab7e-5a301b26e61a","Type":"ContainerDied","Data":"dcd8ec112238bf04e4cd16c37f24f77055ae6bd92869d13874e65b927198f187"}
Jul 17 21:18:28.775794 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.775769    2569 scope.go:117] "RemoveContainer" containerID="5da5ada75a83df885cb17218a663991080abf91f5184b98c7bafc1642f50e699"
Jul 17 21:18:28.776122 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:18:28.776103    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da5ada75a83df885cb17218a663991080abf91f5184b98c7bafc1642f50e699\": container with ID starting with 5da5ada75a83df885cb17218a663991080abf91f5184b98c7bafc1642f50e699 not found: ID does not exist" containerID="5da5ada75a83df885cb17218a663991080abf91f5184b98c7bafc1642f50e699"
Jul 17 21:18:28.776199 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.776130    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da5ada75a83df885cb17218a663991080abf91f5184b98c7bafc1642f50e699"} err="failed to get container status \"5da5ada75a83df885cb17218a663991080abf91f5184b98c7bafc1642f50e699\": rpc error: code = NotFound desc = could not find container \"5da5ada75a83df885cb17218a663991080abf91f5184b98c7bafc1642f50e699\": container with ID starting with 5da5ada75a83df885cb17218a663991080abf91f5184b98c7bafc1642f50e699 not found: ID does not exist"
Jul 17 21:18:28.776199 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.776149    2569 scope.go:117] "RemoveContainer" containerID="c84d8603ae3a8e9cdff9e9e5f71a3c661b1a5678137e2d58e48a47bb335c34ba"
Jul 17 21:18:28.777112 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.776988    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-677f9477cd-7tt7f" podStartSLOduration=1.378015958 podStartE2EDuration="1.776971613s" totalImagesPullingTime="398.955655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:18:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:18:28.774864733 +0000 UTC m=+622.534431432" watchObservedRunningTime="2026-07-17 21:18:28.776971613 +0000 UTC m=+622.536538309"
Jul 17 21:18:28.785171 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.785130    2569 scope.go:117] "RemoveContainer" containerID="c84d8603ae3a8e9cdff9e9e5f71a3c661b1a5678137e2d58e48a47bb335c34ba"
Jul 17 21:18:28.785511 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:18:28.785486    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c84d8603ae3a8e9cdff9e9e5f71a3c661b1a5678137e2d58e48a47bb335c34ba\": container with ID starting with c84d8603ae3a8e9cdff9e9e5f71a3c661b1a5678137e2d58e48a47bb335c34ba not found: ID does not exist" containerID="c84d8603ae3a8e9cdff9e9e5f71a3c661b1a5678137e2d58e48a47bb335c34ba"
Jul 17 21:18:28.785610 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.785520    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c84d8603ae3a8e9cdff9e9e5f71a3c661b1a5678137e2d58e48a47bb335c34ba"} err="failed to get container status \"c84d8603ae3a8e9cdff9e9e5f71a3c661b1a5678137e2d58e48a47bb335c34ba\": rpc error: code = NotFound desc = could not find container \"c84d8603ae3a8e9cdff9e9e5f71a3c661b1a5678137e2d58e48a47bb335c34ba\": container with ID starting with c84d8603ae3a8e9cdff9e9e5f71a3c661b1a5678137e2d58e48a47bb335c34ba not found: ID does not exist"
Jul 17 21:18:28.789599 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.789572    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d4f9f9554-nldt8"]
Jul 17 21:18:28.795477 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.795450    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-d4f9f9554-nldt8"]
Jul 17 21:18:28.807219 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.807195    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2pn7m"]
Jul 17 21:18:28.813784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:28.813753    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2pn7m"]
Jul 17 21:18:30.656707 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:30.656630    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3151c803-ba20-44aa-ab7e-5a301b26e61a" path="/var/lib/kubelet/pods/3151c803-ba20-44aa-ab7e-5a301b26e61a/volumes"
Jul 17 21:18:30.657138 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:30.656945    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50d69fdc-f89d-4a21-87c4-95275d5bae99" path="/var/lib/kubelet/pods/50d69fdc-f89d-4a21-87c4-95275d5bae99/volumes"
Jul 17 21:18:33.340296 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:33.340263    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:18:33.436967 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:33.436936    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:18:33.497858 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:33.497828    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:18:36.744390 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:36.744323    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:18:36.846358 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:18:36.846317    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:08.594253 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:08.594216    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:08.695073 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:08.695036    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:09.059335 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:09.059242    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:09.452899 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:09.452861    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:09.744809 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:09.744714    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:09.838229 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:09.838196    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:09.974180 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:09.974145    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:19:11.750977 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:11.750943    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:21.216045 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:21.216005    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:21.310610 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:21.310570    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:21.940456 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:21.940386    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:22.040339 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:22.040299    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:26.124814 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:26.124781    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:26.517145 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:26.517038    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:19:32.044433 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:32.044382    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:32.722344 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:32.722308    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:19:39.486905 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:39.486865    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:39.548341 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:39.548301    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:42.412284 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:42.412243    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:42.503212 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:42.503175    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:42.572815 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:42.572770    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:43.717065 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:43.717025    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:19:47.426372 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:47.426342    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:49.426327 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:49.426289    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:19:54.676445 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:54.676383    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:54.752949 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:54.752915    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:19:59.448119 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:19:59.448033    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:20:01.638482 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:01.638436    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:20:09.712522 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.712469    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"]
Jul 17 21:20:09.713003 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.712913    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d69fdc-f89d-4a21-87c4-95275d5bae99" containerName="authorino"
Jul 17 21:20:09.713003 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.712926    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d69fdc-f89d-4a21-87c4-95275d5bae99" containerName="authorino"
Jul 17 21:20:09.713003 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.712944    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3151c803-ba20-44aa-ab7e-5a301b26e61a" containerName="authorino"
Jul 17 21:20:09.713003 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.712950    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3151c803-ba20-44aa-ab7e-5a301b26e61a" containerName="authorino"
Jul 17 21:20:09.713232 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.713010    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3151c803-ba20-44aa-ab7e-5a301b26e61a" containerName="authorino"
Jul 17 21:20:09.713232 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.713020    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d69fdc-f89d-4a21-87c4-95275d5bae99" containerName="authorino"
Jul 17 21:20:09.716872 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.716847    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:09.720353 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.720327    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 17 21:20:09.720652 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.720341    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-q6bg6\""
Jul 17 21:20:09.720788 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.720356    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 17 21:20:09.720788 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.720446    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 17 21:20:09.725557 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.725533    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"]
Jul 17 21:20:09.741300 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.741266    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:20:09.773335 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.773304    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114c61c3-3373-4ae2-b9cd-7acc63966e67-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd\" (UID: \"114c61c3-3373-4ae2-b9cd-7acc63966e67\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:09.773537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.773355    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114c61c3-3373-4ae2-b9cd-7acc63966e67-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd\" (UID: \"114c61c3-3373-4ae2-b9cd-7acc63966e67\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:09.773537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.773412    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/114c61c3-3373-4ae2-b9cd-7acc63966e67-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd\" (UID: \"114c61c3-3373-4ae2-b9cd-7acc63966e67\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:09.773537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.773494    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/114c61c3-3373-4ae2-b9cd-7acc63966e67-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd\" (UID: \"114c61c3-3373-4ae2-b9cd-7acc63966e67\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:09.773681 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.773591    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk8t2\" (UniqueName: \"kubernetes.io/projected/114c61c3-3373-4ae2-b9cd-7acc63966e67-kube-api-access-nk8t2\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd\" (UID: \"114c61c3-3373-4ae2-b9cd-7acc63966e67\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:09.773681 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.773664    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114c61c3-3373-4ae2-b9cd-7acc63966e67-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd\" (UID: \"114c61c3-3373-4ae2-b9cd-7acc63966e67\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:09.874715 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.874676    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk8t2\" (UniqueName: \"kubernetes.io/projected/114c61c3-3373-4ae2-b9cd-7acc63966e67-kube-api-access-nk8t2\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd\" (UID: \"114c61c3-3373-4ae2-b9cd-7acc63966e67\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:09.874919 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.874744    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114c61c3-3373-4ae2-b9cd-7acc63966e67-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd\" (UID: \"114c61c3-3373-4ae2-b9cd-7acc63966e67\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:09.874919 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.874792    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114c61c3-3373-4ae2-b9cd-7acc63966e67-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd\" (UID: \"114c61c3-3373-4ae2-b9cd-7acc63966e67\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:09.874919 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.874828    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114c61c3-3373-4ae2-b9cd-7acc63966e67-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd\" (UID: \"114c61c3-3373-4ae2-b9cd-7acc63966e67\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:09.874919 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.874869    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/114c61c3-3373-4ae2-b9cd-7acc63966e67-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd\" (UID: \"114c61c3-3373-4ae2-b9cd-7acc63966e67\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:09.874919 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.874905    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/114c61c3-3373-4ae2-b9cd-7acc63966e67-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd\" (UID: \"114c61c3-3373-4ae2-b9cd-7acc63966e67\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:09.875231 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.875204    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/114c61c3-3373-4ae2-b9cd-7acc63966e67-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd\" (UID: \"114c61c3-3373-4ae2-b9cd-7acc63966e67\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:09.875231 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.875227    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/114c61c3-3373-4ae2-b9cd-7acc63966e67-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd\" (UID: \"114c61c3-3373-4ae2-b9cd-7acc63966e67\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:09.875344 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.875270    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/114c61c3-3373-4ae2-b9cd-7acc63966e67-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd\" (UID: \"114c61c3-3373-4ae2-b9cd-7acc63966e67\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:09.877215 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.877181    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/114c61c3-3373-4ae2-b9cd-7acc63966e67-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd\" (UID: \"114c61c3-3373-4ae2-b9cd-7acc63966e67\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:09.877489 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.877470    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114c61c3-3373-4ae2-b9cd-7acc63966e67-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd\" (UID: \"114c61c3-3373-4ae2-b9cd-7acc63966e67\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:09.881826 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:09.881807    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk8t2\" (UniqueName: \"kubernetes.io/projected/114c61c3-3373-4ae2-b9cd-7acc63966e67-kube-api-access-nk8t2\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd\" (UID: \"114c61c3-3373-4ae2-b9cd-7acc63966e67\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:10.027999 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:10.027907    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:10.170119 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:10.170087    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"]
Jul 17 21:20:10.171090 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:20:10.171060    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod114c61c3_3373_4ae2_b9cd_7acc63966e67.slice/crio-9c03caaca57555f481e7d69193efffd39aee8f10217b943f8e19ca4ec7ac8efb WatchSource:0}: Error finding container 9c03caaca57555f481e7d69193efffd39aee8f10217b943f8e19ca4ec7ac8efb: Status 404 returned error can't find the container with id 9c03caaca57555f481e7d69193efffd39aee8f10217b943f8e19ca4ec7ac8efb
Jul 17 21:20:10.317511 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:10.317423    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd" event={"ID":"114c61c3-3373-4ae2-b9cd-7acc63966e67","Type":"ContainerStarted","Data":"9c03caaca57555f481e7d69193efffd39aee8f10217b943f8e19ca4ec7ac8efb"}
Jul 17 21:20:12.320280 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:12.320239    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:20:12.330657 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:12.330620    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd" event={"ID":"114c61c3-3373-4ae2-b9cd-7acc63966e67","Type":"ContainerStarted","Data":"11a54a7f0850657de84559b85889d395aaeee90896905d52dfc6cf818ccb7911"}
Jul 17 21:20:12.348453 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:12.348389    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd" podStartSLOduration=2.19844009 podStartE2EDuration="3.348371754s" totalImagesPullingTime="1.149931664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:20:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:20:12.346253662 +0000 UTC m=+726.105820351" watchObservedRunningTime="2026-07-17 21:20:12.348371754 +0000 UTC m=+726.107938447"
Jul 17 21:20:20.028085 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:20.028049    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:20.028561 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:20.028124    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:20.040514 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:20.040488    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:20.386658 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:20.386621    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd"
Jul 17 21:20:36.814360 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:20:36.814312    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:21:29.113484 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:29.113444    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84b6ff4847-rd2kz"]
Jul 17 21:21:29.117214 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:29.117188    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84b6ff4847-rd2kz"
Jul 17 21:21:29.122939 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:29.122910    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84b6ff4847-rd2kz"]
Jul 17 21:21:29.265275 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:29.265239    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e390f181-7b81-4dc9-96f4-a94056ecdbdc-tls-cert\") pod \"authorino-84b6ff4847-rd2kz\" (UID: \"e390f181-7b81-4dc9-96f4-a94056ecdbdc\") " pod="kuadrant-system/authorino-84b6ff4847-rd2kz"
Jul 17 21:21:29.265473 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:29.265283    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smcd2\" (UniqueName: \"kubernetes.io/projected/e390f181-7b81-4dc9-96f4-a94056ecdbdc-kube-api-access-smcd2\") pod \"authorino-84b6ff4847-rd2kz\" (UID: \"e390f181-7b81-4dc9-96f4-a94056ecdbdc\") " pod="kuadrant-system/authorino-84b6ff4847-rd2kz"
Jul 17 21:21:29.366066 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:29.365952    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smcd2\" (UniqueName: \"kubernetes.io/projected/e390f181-7b81-4dc9-96f4-a94056ecdbdc-kube-api-access-smcd2\") pod \"authorino-84b6ff4847-rd2kz\" (UID: \"e390f181-7b81-4dc9-96f4-a94056ecdbdc\") " pod="kuadrant-system/authorino-84b6ff4847-rd2kz"
Jul 17 21:21:29.366066 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:29.366057    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e390f181-7b81-4dc9-96f4-a94056ecdbdc-tls-cert\") pod \"authorino-84b6ff4847-rd2kz\" (UID: \"e390f181-7b81-4dc9-96f4-a94056ecdbdc\") " pod="kuadrant-system/authorino-84b6ff4847-rd2kz"
Jul 17 21:21:29.368541 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:29.368507    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e390f181-7b81-4dc9-96f4-a94056ecdbdc-tls-cert\") pod \"authorino-84b6ff4847-rd2kz\" (UID: \"e390f181-7b81-4dc9-96f4-a94056ecdbdc\") " pod="kuadrant-system/authorino-84b6ff4847-rd2kz"
Jul 17 21:21:29.373607 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:29.373582    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smcd2\" (UniqueName: \"kubernetes.io/projected/e390f181-7b81-4dc9-96f4-a94056ecdbdc-kube-api-access-smcd2\") pod \"authorino-84b6ff4847-rd2kz\" (UID: \"e390f181-7b81-4dc9-96f4-a94056ecdbdc\") " pod="kuadrant-system/authorino-84b6ff4847-rd2kz"
Jul 17 21:21:29.427417 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:29.427379    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84b6ff4847-rd2kz"
Jul 17 21:21:29.560716 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:29.560689    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84b6ff4847-rd2kz"]
Jul 17 21:21:29.563029 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:21:29.562998    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode390f181_7b81_4dc9_96f4_a94056ecdbdc.slice/crio-21ff5c2a09bc21ac6838f8f207794f99f8a27c2a9de5663e4d260bb1e5813471 WatchSource:0}: Error finding container 21ff5c2a09bc21ac6838f8f207794f99f8a27c2a9de5663e4d260bb1e5813471: Status 404 returned error can't find the container with id 21ff5c2a09bc21ac6838f8f207794f99f8a27c2a9de5663e4d260bb1e5813471
Jul 17 21:21:29.757229 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:29.757146    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84b6ff4847-rd2kz" event={"ID":"e390f181-7b81-4dc9-96f4-a94056ecdbdc","Type":"ContainerStarted","Data":"21ff5c2a09bc21ac6838f8f207794f99f8a27c2a9de5663e4d260bb1e5813471"}
Jul 17 21:21:30.765691 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:30.765593    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84b6ff4847-rd2kz" event={"ID":"e390f181-7b81-4dc9-96f4-a94056ecdbdc","Type":"ContainerStarted","Data":"7f1a7a49853e7d80d6cdfd21446c0518cd5f0ed1e1f5b4627515efeecd3ed2b5"}
Jul 17 21:21:30.779922 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:30.779860    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-84b6ff4847-rd2kz" podStartSLOduration=0.932058758 podStartE2EDuration="1.77983661s" totalImagesPullingTime="847.777852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:21:30.778519385 +0000 UTC m=+804.538086081" watchObservedRunningTime="2026-07-17 21:21:30.77983661 +0000 UTC m=+804.539403305"
Jul 17 21:21:30.831110 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:30.831073    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-677f9477cd-7tt7f"]
Jul 17 21:21:30.831331 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:30.831305    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-677f9477cd-7tt7f" podUID="7e338ed9-bdc1-41cc-b12b-998cc168011e" containerName="authorino" containerID="cri-o://ce0050cd0694a79df8b8e766502a7cd59e22bc773daf6a9adb20a59c4fb217ab" gracePeriod=30
Jul 17 21:21:31.086530 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:31.086500    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-677f9477cd-7tt7f"
Jul 17 21:21:31.183558 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:31.183525    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2cqh\" (UniqueName: \"kubernetes.io/projected/7e338ed9-bdc1-41cc-b12b-998cc168011e-kube-api-access-s2cqh\") pod \"7e338ed9-bdc1-41cc-b12b-998cc168011e\" (UID: \"7e338ed9-bdc1-41cc-b12b-998cc168011e\") "
Jul 17 21:21:31.183745 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:31.183618    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e338ed9-bdc1-41cc-b12b-998cc168011e-tls-cert\") pod \"7e338ed9-bdc1-41cc-b12b-998cc168011e\" (UID: \"7e338ed9-bdc1-41cc-b12b-998cc168011e\") "
Jul 17 21:21:31.185684 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:31.185649    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e338ed9-bdc1-41cc-b12b-998cc168011e-kube-api-access-s2cqh" (OuterVolumeSpecName: "kube-api-access-s2cqh") pod "7e338ed9-bdc1-41cc-b12b-998cc168011e" (UID: "7e338ed9-bdc1-41cc-b12b-998cc168011e"). InnerVolumeSpecName "kube-api-access-s2cqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:21:31.196147 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:31.196106    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e338ed9-bdc1-41cc-b12b-998cc168011e-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "7e338ed9-bdc1-41cc-b12b-998cc168011e" (UID: "7e338ed9-bdc1-41cc-b12b-998cc168011e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:21:31.284758 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:31.284720    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2cqh\" (UniqueName: \"kubernetes.io/projected/7e338ed9-bdc1-41cc-b12b-998cc168011e-kube-api-access-s2cqh\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:21:31.284758 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:31.284752    2569 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e338ed9-bdc1-41cc-b12b-998cc168011e-tls-cert\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:21:31.370844 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:31.370795    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:21:31.466997 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:31.466960    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:21:31.774350 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:31.774249    2569 generic.go:356] "Generic (PLEG): container finished" podID="7e338ed9-bdc1-41cc-b12b-998cc168011e" containerID="ce0050cd0694a79df8b8e766502a7cd59e22bc773daf6a9adb20a59c4fb217ab" exitCode=0
Jul 17 21:21:31.774350 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:31.774315    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-677f9477cd-7tt7f"
Jul 17 21:21:31.774888 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:31.774310    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-677f9477cd-7tt7f" event={"ID":"7e338ed9-bdc1-41cc-b12b-998cc168011e","Type":"ContainerDied","Data":"ce0050cd0694a79df8b8e766502a7cd59e22bc773daf6a9adb20a59c4fb217ab"}
Jul 17 21:21:31.774888 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:31.774444    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-677f9477cd-7tt7f" event={"ID":"7e338ed9-bdc1-41cc-b12b-998cc168011e","Type":"ContainerDied","Data":"fe15bb63cfbaa16efedd7228dc05e97d68a37558913d37898cd2df24d7796c76"}
Jul 17 21:21:31.774888 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:31.774473    2569 scope.go:117] "RemoveContainer" containerID="ce0050cd0694a79df8b8e766502a7cd59e22bc773daf6a9adb20a59c4fb217ab"
Jul 17 21:21:31.784826 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:31.784806    2569 scope.go:117] "RemoveContainer" containerID="ce0050cd0694a79df8b8e766502a7cd59e22bc773daf6a9adb20a59c4fb217ab"
Jul 17 21:21:31.785101 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:21:31.785082    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce0050cd0694a79df8b8e766502a7cd59e22bc773daf6a9adb20a59c4fb217ab\": container with ID starting with ce0050cd0694a79df8b8e766502a7cd59e22bc773daf6a9adb20a59c4fb217ab not found: ID does not exist" containerID="ce0050cd0694a79df8b8e766502a7cd59e22bc773daf6a9adb20a59c4fb217ab"
Jul 17 21:21:31.785154 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:31.785109    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0050cd0694a79df8b8e766502a7cd59e22bc773daf6a9adb20a59c4fb217ab"} err="failed to get container status \"ce0050cd0694a79df8b8e766502a7cd59e22bc773daf6a9adb20a59c4fb217ab\": rpc error: code = NotFound desc = could not find container \"ce0050cd0694a79df8b8e766502a7cd59e22bc773daf6a9adb20a59c4fb217ab\": container with ID starting with ce0050cd0694a79df8b8e766502a7cd59e22bc773daf6a9adb20a59c4fb217ab not found: ID does not exist"
Jul 17 21:21:31.798658 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:31.798607    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-677f9477cd-7tt7f"]
Jul 17 21:21:31.800863 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:31.800844    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-677f9477cd-7tt7f"]
Jul 17 21:21:32.568086 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:32.568053    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:21:32.657539 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:32.657503    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e338ed9-bdc1-41cc-b12b-998cc168011e" path="/var/lib/kubelet/pods/7e338ed9-bdc1-41cc-b12b-998cc168011e/volumes"
Jul 17 21:21:32.677904 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:32.677855    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:21:39.146074 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:39.146031    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:21:59.421869 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:21:59.421830    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:22:10.110488 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:22:10.110447    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:22:20.521157 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:22:20.521109    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:23:06.641589 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:23:06.641509    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:23:06.643446 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:23:06.643420    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:23:10.431665 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:23:10.431621    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:23:36.016013 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:23:36.015975    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:23:37.286010 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:23:37.285967    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:23:58.931341 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:23:58.931304    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:24:12.028670 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:24:12.028628    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:24:17.677969 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:24:17.677929    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:24:17.770683 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:24:17.770648    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:24:17.877564 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:24:17.877520    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:24:23.678494 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:24:23.678445    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:24:23.835894 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:24:23.835858    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:24:23.935231 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:24:23.935142    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:24:24.034368 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:24:24.034329    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:24:29.858108 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:24:29.858023    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:24:29.964140 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:24:29.964097    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:24:35.933879 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:24:35.933841    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:24:36.031216 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:24:36.031178    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:24:47.015008 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:24:47.014971    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:24:56.930254 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:24:56.930221    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:25:07.247765 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:25:07.247727    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:25:15.128675 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:25:15.128635    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:25:27.132728 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:25:27.132690    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:25:38.627598 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:25:38.627556    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:25:49.629081 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:25:49.629041    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:25:59.723189 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:25:59.723106    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:26:21.421417 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:26:21.421362    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:26:32.521348 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:26:32.521314    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:26:56.724969 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:26:56.724930    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:27:11.456312 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:27:11.456272    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:27:22.725666 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:27:22.725622    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:27:30.631510 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:27:30.631418    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:27:30.721442 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:27:30.721378    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:27:32.421871 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:27:32.421837    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:27:42.420610 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:27:42.420572    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:27:52.013088 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:27:52.013050    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:28:03.418601 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:03.418561    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:28:06.678654 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:06.678624    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:28:06.695853 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:06.695823    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:28:27.169817 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:27.169778    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g"]
Jul 17 21:28:27.170441 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:27.170134    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g" podUID="6a39633e-78fe-434f-9263-b311f3a52c1b" containerName="manager" containerID="cri-o://48d380e2bfb60ff8a21ffab15037d7f5919160c86f846d015d7c047359ea5768" gracePeriod=10
Jul 17 21:28:29.623474 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:29.623452    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g"
Jul 17 21:28:29.742733 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:29.742644    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhx4l\" (UniqueName: \"kubernetes.io/projected/6a39633e-78fe-434f-9263-b311f3a52c1b-kube-api-access-hhx4l\") pod \"6a39633e-78fe-434f-9263-b311f3a52c1b\" (UID: \"6a39633e-78fe-434f-9263-b311f3a52c1b\") "
Jul 17 21:28:29.742733 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:29.742687    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a39633e-78fe-434f-9263-b311f3a52c1b-extensions-socket-volume\") pod \"6a39633e-78fe-434f-9263-b311f3a52c1b\" (UID: \"6a39633e-78fe-434f-9263-b311f3a52c1b\") "
Jul 17 21:28:29.743134 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:29.743108    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a39633e-78fe-434f-9263-b311f3a52c1b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6a39633e-78fe-434f-9263-b311f3a52c1b" (UID: "6a39633e-78fe-434f-9263-b311f3a52c1b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:28:29.744703 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:29.744680    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a39633e-78fe-434f-9263-b311f3a52c1b-kube-api-access-hhx4l" (OuterVolumeSpecName: "kube-api-access-hhx4l") pod "6a39633e-78fe-434f-9263-b311f3a52c1b" (UID: "6a39633e-78fe-434f-9263-b311f3a52c1b"). InnerVolumeSpecName "kube-api-access-hhx4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:28:29.843984 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:29.843944    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhx4l\" (UniqueName: \"kubernetes.io/projected/6a39633e-78fe-434f-9263-b311f3a52c1b-kube-api-access-hhx4l\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:28:29.843984 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:29.843975    2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a39633e-78fe-434f-9263-b311f3a52c1b-extensions-socket-volume\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:28:30.160565 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:30.160528    2569 generic.go:356] "Generic (PLEG): container finished" podID="6a39633e-78fe-434f-9263-b311f3a52c1b" containerID="48d380e2bfb60ff8a21ffab15037d7f5919160c86f846d015d7c047359ea5768" exitCode=0
Jul 17 21:28:30.160763 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:30.160583    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g"
Jul 17 21:28:30.160763 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:30.160592    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g" event={"ID":"6a39633e-78fe-434f-9263-b311f3a52c1b","Type":"ContainerDied","Data":"48d380e2bfb60ff8a21ffab15037d7f5919160c86f846d015d7c047359ea5768"}
Jul 17 21:28:30.160763 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:30.160644    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g" event={"ID":"6a39633e-78fe-434f-9263-b311f3a52c1b","Type":"ContainerDied","Data":"60f3cbf3623732cc3fcbab8c1ecb9eb86e04678365eb5685c7a91a42fae7595e"}
Jul 17 21:28:30.160763 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:30.160672    2569 scope.go:117] "RemoveContainer" containerID="48d380e2bfb60ff8a21ffab15037d7f5919160c86f846d015d7c047359ea5768"
Jul 17 21:28:30.171625 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:30.171608    2569 scope.go:117] "RemoveContainer" containerID="48d380e2bfb60ff8a21ffab15037d7f5919160c86f846d015d7c047359ea5768"
Jul 17 21:28:30.171894 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:28:30.171872    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d380e2bfb60ff8a21ffab15037d7f5919160c86f846d015d7c047359ea5768\": container with ID starting with 48d380e2bfb60ff8a21ffab15037d7f5919160c86f846d015d7c047359ea5768 not found: ID does not exist" containerID="48d380e2bfb60ff8a21ffab15037d7f5919160c86f846d015d7c047359ea5768"
Jul 17 21:28:30.171941 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:30.171903    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d380e2bfb60ff8a21ffab15037d7f5919160c86f846d015d7c047359ea5768"} err="failed to get container status \"48d380e2bfb60ff8a21ffab15037d7f5919160c86f846d015d7c047359ea5768\": rpc error: code = NotFound desc = could not find container \"48d380e2bfb60ff8a21ffab15037d7f5919160c86f846d015d7c047359ea5768\": container with ID starting with 48d380e2bfb60ff8a21ffab15037d7f5919160c86f846d015d7c047359ea5768 not found: ID does not exist"
Jul 17 21:28:30.186117 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:30.186096    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g"]
Jul 17 21:28:30.189130 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:30.189103    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqh4g"]
Jul 17 21:28:30.656093 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:28:30.656059    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a39633e-78fe-434f-9263-b311f3a52c1b" path="/var/lib/kubelet/pods/6a39633e-78fe-434f-9263-b311f3a52c1b/volumes"
Jul 17 21:29:16.105920 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:16.105816    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:29:28.378211 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:28.378174    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n42th"]
Jul 17 21:29:28.378633 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:28.378585    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a39633e-78fe-434f-9263-b311f3a52c1b" containerName="manager"
Jul 17 21:29:28.378633 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:28.378599    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a39633e-78fe-434f-9263-b311f3a52c1b" containerName="manager"
Jul 17 21:29:28.378633 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:28.378614    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e338ed9-bdc1-41cc-b12b-998cc168011e" containerName="authorino"
Jul 17 21:29:28.378633 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:28.378620    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e338ed9-bdc1-41cc-b12b-998cc168011e" containerName="authorino"
Jul 17 21:29:28.378774 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:28.378680    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a39633e-78fe-434f-9263-b311f3a52c1b" containerName="manager"
Jul 17 21:29:28.378774 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:28.378697    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e338ed9-bdc1-41cc-b12b-998cc168011e" containerName="authorino"
Jul 17 21:29:28.382711 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:28.382694    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n42th"
Jul 17 21:29:28.385965 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:28.385946    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tp7kt\""
Jul 17 21:29:28.400621 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:28.400594    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n42th"]
Jul 17 21:29:28.545324 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:28.545288    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5248760-056f-4372-8617-4b4c22687016-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-n42th\" (UID: \"e5248760-056f-4372-8617-4b4c22687016\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n42th"
Jul 17 21:29:28.545528 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:28.545448    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4dx2\" (UniqueName: \"kubernetes.io/projected/e5248760-056f-4372-8617-4b4c22687016-kube-api-access-g4dx2\") pod \"kuadrant-operator-controller-manager-86878c769f-n42th\" (UID: \"e5248760-056f-4372-8617-4b4c22687016\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n42th"
Jul 17 21:29:28.646241 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:28.646140    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4dx2\" (UniqueName: \"kubernetes.io/projected/e5248760-056f-4372-8617-4b4c22687016-kube-api-access-g4dx2\") pod \"kuadrant-operator-controller-manager-86878c769f-n42th\" (UID: \"e5248760-056f-4372-8617-4b4c22687016\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n42th"
Jul 17 21:29:28.646241 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:28.646210    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5248760-056f-4372-8617-4b4c22687016-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-n42th\" (UID: \"e5248760-056f-4372-8617-4b4c22687016\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n42th"
Jul 17 21:29:28.646664 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:28.646643    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5248760-056f-4372-8617-4b4c22687016-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-n42th\" (UID: \"e5248760-056f-4372-8617-4b4c22687016\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n42th"
Jul 17 21:29:28.654132 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:28.654107    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4dx2\" (UniqueName: \"kubernetes.io/projected/e5248760-056f-4372-8617-4b4c22687016-kube-api-access-g4dx2\") pod \"kuadrant-operator-controller-manager-86878c769f-n42th\" (UID: \"e5248760-056f-4372-8617-4b4c22687016\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n42th"
Jul 17 21:29:28.692428 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:28.692355    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n42th"
Jul 17 21:29:28.832666 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:28.832635    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n42th"]
Jul 17 21:29:28.834119 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:29:28.834092    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5248760_056f_4372_8617_4b4c22687016.slice/crio-9b60c896a4d64dce19aa4090b6c11be859784e27e7992cdaa68875563c98ea30 WatchSource:0}: Error finding container 9b60c896a4d64dce19aa4090b6c11be859784e27e7992cdaa68875563c98ea30: Status 404 returned error can't find the container with id 9b60c896a4d64dce19aa4090b6c11be859784e27e7992cdaa68875563c98ea30
Jul 17 21:29:29.478555 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:29.478515    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n42th" event={"ID":"e5248760-056f-4372-8617-4b4c22687016","Type":"ContainerStarted","Data":"46607111842cb946341efe5672ae074fe1a08f37ef01d33c5a7e748232024f43"}
Jul 17 21:29:29.479030 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:29.478597    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n42th" event={"ID":"e5248760-056f-4372-8617-4b4c22687016","Type":"ContainerStarted","Data":"9b60c896a4d64dce19aa4090b6c11be859784e27e7992cdaa68875563c98ea30"}
Jul 17 21:29:29.479030 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:29.478633    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n42th"
Jul 17 21:29:29.496468 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:29.496396    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n42th" podStartSLOduration=1.4963821560000001 podStartE2EDuration="1.496382156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:29:29.494652603 +0000 UTC m=+1283.254219294" watchObservedRunningTime="2026-07-17 21:29:29.496382156 +0000 UTC m=+1283.255948850"
Jul 17 21:29:40.486534 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:40.486503    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n42th"
Jul 17 21:29:48.721786 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:29:48.721754    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:30:00.138832 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:00.138796    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738730-qqw5r"]
Jul 17 21:30:00.143995 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:00.143975    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738730-qqw5r"
Jul 17 21:30:00.146243 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:00.146215    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:30:00.148248 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:00.148224    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738730-qqw5r"]
Jul 17 21:30:00.155391 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:00.155369    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738730-qqw5r"
Jul 17 21:30:00.292175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:00.292148    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738730-qqw5r"]
Jul 17 21:30:00.294348 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:30:00.294322    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817f33cb_969d_467c_8b7b_36215e895153.slice/crio-811f310067237536943f930e98ebcdce1a7ca993c038539c0bdac16ef0d18992 WatchSource:0}: Error finding container 811f310067237536943f930e98ebcdce1a7ca993c038539c0bdac16ef0d18992: Status 404 returned error can't find the container with id 811f310067237536943f930e98ebcdce1a7ca993c038539c0bdac16ef0d18992
Jul 17 21:30:00.296218 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:00.296199    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 21:30:00.657059 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:00.657026    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738730-qqw5r" event={"ID":"817f33cb-969d-467c-8b7b-36215e895153","Type":"ContainerStarted","Data":"811f310067237536943f930e98ebcdce1a7ca993c038539c0bdac16ef0d18992"}
Jul 17 21:30:00.810143 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:00.810096    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:30:08.712479 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:08.712441    2569 generic.go:356] "Generic (PLEG): container finished" podID="817f33cb-969d-467c-8b7b-36215e895153" containerID="58e73c8af80d8dfe2b32c9cde26c6d12962a972f3d33da5cf59d2ee0d76b56d2" exitCode=0
Jul 17 21:30:08.712879 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:08.712525    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738730-qqw5r" event={"ID":"817f33cb-969d-467c-8b7b-36215e895153","Type":"ContainerDied","Data":"58e73c8af80d8dfe2b32c9cde26c6d12962a972f3d33da5cf59d2ee0d76b56d2"}
Jul 17 21:30:09.719929 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:09.719893    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:30:09.878943 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:09.878911    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738730-qqw5r"
Jul 17 21:30:10.727093 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:10.727066    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738730-qqw5r"
Jul 17 21:30:10.727093 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:10.727076    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738730-qqw5r" event={"ID":"817f33cb-969d-467c-8b7b-36215e895153","Type":"ContainerDied","Data":"811f310067237536943f930e98ebcdce1a7ca993c038539c0bdac16ef0d18992"}
Jul 17 21:30:10.727550 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:10.727131    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="811f310067237536943f930e98ebcdce1a7ca993c038539c0bdac16ef0d18992"
Jul 17 21:30:23.222476 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:23.222438    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:30:33.114324 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:33.114241    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:30:42.328172 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:42.328135    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:30:45.050248 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:45.050208    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:30:51.417524 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:30:51.417489    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:31:02.515386 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:31:02.515322    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:31:03.967765 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:31:03.967731    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:31:04.068745 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:31:04.068695    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:31:14.411593 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:31:14.411556    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:31:22.319010 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:31:22.318970    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:31:34.731024 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:31:34.730983    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:31:58.336877 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:31:58.336843    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:32:06.235965 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:32:06.235871    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:32:19.390805 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:32:19.390762    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:32:33.352855 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:32:33.352809    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:32:53.020924 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:32:53.020886    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:33:06.716374 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:33:06.716339    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:33:06.738859 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:33:06.738826    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:33:08.554334 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:33:08.554287    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:33:14.351554 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:33:14.351509    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:33:14.436558 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:33:14.436523    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:33:18.207534 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:33:18.207496    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:33:31.547289 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:33:31.547246    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:33:41.027875 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:33:41.027834    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:33:54.133541 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:33:54.133484    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:34:03.423717 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:34:03.423664    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:34:13.425907 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:34:13.425866    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:34:26.914579 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:34:26.914545    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:34:38.151112 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:34:38.151074    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:34:51.417580 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:34:51.417544    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:35:01.108278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:35:01.108198    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:35:18.317968 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:35:18.317933    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:35:31.417117 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:35:31.417078    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:35:33.623728 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:35:33.623686    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:35:41.036471 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:35:41.036438    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:35:44.815861 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:35:44.815828    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:35:49.311364 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:35:49.311322    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:35:54.115715 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:35:54.115673    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:35:54.386501 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:35:54.386397    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:35:57.228706 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:35:57.228660    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:35:57.331032 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:35:57.330994    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:35:57.668476 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:35:57.668439    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:36:03.309588 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:03.309554    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:36:03.414124 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:03.414083    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:36:03.598813 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:03.598779    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:36:03.718593 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:03.718559    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:36:04.017019 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:04.016933    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:36:04.726063 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:04.726025    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:36:04.812927 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:04.812869    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:36:04.886562 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:04.886532    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:36:05.348993 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:05.348959    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:36:05.450211 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:05.450178    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:36:05.551270 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:05.551222    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:36:08.143314 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:08.143276    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:36:09.673450 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:09.673396    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w"]
Jul 17 21:36:09.673839 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:09.673826    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="817f33cb-969d-467c-8b7b-36215e895153" containerName="cleanup"
Jul 17 21:36:09.673884 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:09.673840    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="817f33cb-969d-467c-8b7b-36215e895153" containerName="cleanup"
Jul 17 21:36:09.673933 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:09.673923    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="817f33cb-969d-467c-8b7b-36215e895153" containerName="cleanup"
Jul 17 21:36:09.676906 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:09.676888    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w"
Jul 17 21:36:09.678965 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:09.678945    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:36:09.679076 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:09.678944    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:36:09.683880 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:09.683856    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w"]
Jul 17 21:36:09.738488 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:09.738454    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a406f1-237a-40fc-b10e-247f5cfa34d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-5cd4w\" (UID: \"74a406f1-237a-40fc-b10e-247f5cfa34d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w"
Jul 17 21:36:09.839294 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:09.839257    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a406f1-237a-40fc-b10e-247f5cfa34d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-5cd4w\" (UID: \"74a406f1-237a-40fc-b10e-247f5cfa34d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w"
Jul 17 21:36:09.839946 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:09.839922    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a406f1-237a-40fc-b10e-247f5cfa34d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-5cd4w\" (UID: \"74a406f1-237a-40fc-b10e-247f5cfa34d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w"
Jul 17 21:36:09.989806 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:09.989713    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w"
Jul 17 21:36:10.126258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:10.126233    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w"]
Jul 17 21:36:10.128366 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:36:10.128339    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a406f1_237a_40fc_b10e_247f5cfa34d4.slice/crio-a628ba32191236cfc92bc3815f82b30aa522b164ef605f21229cfb3155dc3fe3 WatchSource:0}: Error finding container a628ba32191236cfc92bc3815f82b30aa522b164ef605f21229cfb3155dc3fe3: Status 404 returned error can't find the container with id a628ba32191236cfc92bc3815f82b30aa522b164ef605f21229cfb3155dc3fe3
Jul 17 21:36:10.804998 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:10.804911    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w" event={"ID":"74a406f1-237a-40fc-b10e-247f5cfa34d4","Type":"ContainerStarted","Data":"b45a65a26b235f86d3aab9e0a255390b8d74e1af02207c8ea8349bae3e10a77b"}
Jul 17 21:36:10.804998 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:10.804961    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w" event={"ID":"74a406f1-237a-40fc-b10e-247f5cfa34d4","Type":"ContainerStarted","Data":"a628ba32191236cfc92bc3815f82b30aa522b164ef605f21229cfb3155dc3fe3"}
Jul 17 21:36:10.821245 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:10.821195    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w" podStartSLOduration=1.821179833 podStartE2EDuration="1.821179833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:36:10.820180618 +0000 UTC m=+1684.579747311" watchObservedRunningTime="2026-07-17 21:36:10.821179833 +0000 UTC m=+1684.580746526"
Jul 17 21:36:11.813793 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:11.813755    2569 generic.go:356] "Generic (PLEG): container finished" podID="74a406f1-237a-40fc-b10e-247f5cfa34d4" containerID="b45a65a26b235f86d3aab9e0a255390b8d74e1af02207c8ea8349bae3e10a77b" exitCode=7
Jul 17 21:36:11.814221 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:11.813846    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w" event={"ID":"74a406f1-237a-40fc-b10e-247f5cfa34d4","Type":"ContainerDied","Data":"b45a65a26b235f86d3aab9e0a255390b8d74e1af02207c8ea8349bae3e10a77b"}
Jul 17 21:36:11.814221 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:11.813984    2569 scope.go:117] "RemoveContainer" containerID="b45a65a26b235f86d3aab9e0a255390b8d74e1af02207c8ea8349bae3e10a77b"
Jul 17 21:36:12.822247 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:12.822143    2569 generic.go:356] "Generic (PLEG): container finished" podID="74a406f1-237a-40fc-b10e-247f5cfa34d4" containerID="18982d3c2d1ded58160f5039c2c0321a0a77f2f403ec8f3ef15a0b955a4725d6" exitCode=7
Jul 17 21:36:12.822247 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:12.822217    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w" event={"ID":"74a406f1-237a-40fc-b10e-247f5cfa34d4","Type":"ContainerDied","Data":"18982d3c2d1ded58160f5039c2c0321a0a77f2f403ec8f3ef15a0b955a4725d6"}
Jul 17 21:36:12.822938 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:12.822293    2569 scope.go:117] "RemoveContainer" containerID="b45a65a26b235f86d3aab9e0a255390b8d74e1af02207c8ea8349bae3e10a77b"
Jul 17 21:36:12.822938 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:12.822559    2569 scope.go:117] "RemoveContainer" containerID="18982d3c2d1ded58160f5039c2c0321a0a77f2f403ec8f3ef15a0b955a4725d6"
Jul 17 21:36:12.822938 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:36:12.822778    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-5cd4w_odh-ai-gateway-infra(74a406f1-237a-40fc-b10e-247f5cfa34d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w" podUID="74a406f1-237a-40fc-b10e-247f5cfa34d4"
Jul 17 21:36:13.829918 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:13.829863    2569 scope.go:117] "RemoveContainer" containerID="18982d3c2d1ded58160f5039c2c0321a0a77f2f403ec8f3ef15a0b955a4725d6"
Jul 17 21:36:13.830289 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:36:13.830043    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-5cd4w_odh-ai-gateway-infra(74a406f1-237a-40fc-b10e-247f5cfa34d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w" podUID="74a406f1-237a-40fc-b10e-247f5cfa34d4"
Jul 17 21:36:15.508309 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:15.508273    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:36:25.086125 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:25.086078    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:36:26.655239 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:26.655209    2569 scope.go:117] "RemoveContainer" containerID="18982d3c2d1ded58160f5039c2c0321a0a77f2f403ec8f3ef15a0b955a4725d6"
Jul 17 21:36:26.905727 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:26.905644    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w" event={"ID":"74a406f1-237a-40fc-b10e-247f5cfa34d4","Type":"ContainerStarted","Data":"db40f5414a7b5105e9e1de7e1b9f4ff2f2d73daccbcaff25ef976c67af82539c"}
Jul 17 21:36:27.913560 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:27.913525    2569 generic.go:356] "Generic (PLEG): container finished" podID="74a406f1-237a-40fc-b10e-247f5cfa34d4" containerID="db40f5414a7b5105e9e1de7e1b9f4ff2f2d73daccbcaff25ef976c67af82539c" exitCode=7
Jul 17 21:36:27.913560 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:27.913563    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w" event={"ID":"74a406f1-237a-40fc-b10e-247f5cfa34d4","Type":"ContainerDied","Data":"db40f5414a7b5105e9e1de7e1b9f4ff2f2d73daccbcaff25ef976c67af82539c"}
Jul 17 21:36:27.914101 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:27.913611    2569 scope.go:117] "RemoveContainer" containerID="18982d3c2d1ded58160f5039c2c0321a0a77f2f403ec8f3ef15a0b955a4725d6"
Jul 17 21:36:27.914101 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:27.913916    2569 scope.go:117] "RemoveContainer" containerID="db40f5414a7b5105e9e1de7e1b9f4ff2f2d73daccbcaff25ef976c67af82539c"
Jul 17 21:36:27.914211 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:36:27.914168    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-5cd4w_odh-ai-gateway-infra(74a406f1-237a-40fc-b10e-247f5cfa34d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w" podUID="74a406f1-237a-40fc-b10e-247f5cfa34d4"
Jul 17 21:36:27.932945 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:27.932919    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w"]
Jul 17 21:36:29.063900 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:29.063870    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w"
Jul 17 21:36:29.215809 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:29.215709    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a406f1-237a-40fc-b10e-247f5cfa34d4-maas-api-service-ca\") pod \"74a406f1-237a-40fc-b10e-247f5cfa34d4\" (UID: \"74a406f1-237a-40fc-b10e-247f5cfa34d4\") "
Jul 17 21:36:29.216458 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:29.216373    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a406f1-237a-40fc-b10e-247f5cfa34d4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74a406f1-237a-40fc-b10e-247f5cfa34d4" (UID: "74a406f1-237a-40fc-b10e-247f5cfa34d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:36:29.317049 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:29.316948    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a406f1-237a-40fc-b10e-247f5cfa34d4-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:36:29.927467 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:29.927432    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w" event={"ID":"74a406f1-237a-40fc-b10e-247f5cfa34d4","Type":"ContainerDied","Data":"a628ba32191236cfc92bc3815f82b30aa522b164ef605f21229cfb3155dc3fe3"}
Jul 17 21:36:29.927467 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:29.927466    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w"
Jul 17 21:36:29.927690 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:29.927496    2569 scope.go:117] "RemoveContainer" containerID="db40f5414a7b5105e9e1de7e1b9f4ff2f2d73daccbcaff25ef976c67af82539c"
Jul 17 21:36:29.951100 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:29.951071    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w"]
Jul 17 21:36:29.956964 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:29.956937    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5cd4w"]
Jul 17 21:36:30.656998 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:36:30.656965    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74a406f1-237a-40fc-b10e-247f5cfa34d4" path="/var/lib/kubelet/pods/74a406f1-237a-40fc-b10e-247f5cfa34d4/volumes"
Jul 17 21:37:09.720389 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:09.720352    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp"]
Jul 17 21:37:09.720885 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:09.720747    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74a406f1-237a-40fc-b10e-247f5cfa34d4" containerName="revoke-keys"
Jul 17 21:37:09.720885 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:09.720759    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="74a406f1-237a-40fc-b10e-247f5cfa34d4" containerName="revoke-keys"
Jul 17 21:37:09.720885 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:09.720770    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74a406f1-237a-40fc-b10e-247f5cfa34d4" containerName="revoke-keys"
Jul 17 21:37:09.720885 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:09.720776    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="74a406f1-237a-40fc-b10e-247f5cfa34d4" containerName="revoke-keys"
Jul 17 21:37:09.720885 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:09.720846    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="74a406f1-237a-40fc-b10e-247f5cfa34d4" containerName="revoke-keys"
Jul 17 21:37:09.720885 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:09.720857    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="74a406f1-237a-40fc-b10e-247f5cfa34d4" containerName="revoke-keys"
Jul 17 21:37:09.724109 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:09.724087    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp"
Jul 17 21:37:09.727093 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:09.727049    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:37:09.727385 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:09.727361    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:37:09.733113 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:09.733088    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp"]
Jul 17 21:37:09.864297 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:09.864259    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b28c99df-8560-4b39-9a1e-574aecf20e82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-w46qp\" (UID: \"b28c99df-8560-4b39-9a1e-574aecf20e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp"
Jul 17 21:37:09.965175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:09.965136    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b28c99df-8560-4b39-9a1e-574aecf20e82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-w46qp\" (UID: \"b28c99df-8560-4b39-9a1e-574aecf20e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp"
Jul 17 21:37:09.965869 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:09.965848    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b28c99df-8560-4b39-9a1e-574aecf20e82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-w46qp\" (UID: \"b28c99df-8560-4b39-9a1e-574aecf20e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp"
Jul 17 21:37:10.037113 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:10.037029    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp"
Jul 17 21:37:10.180925 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:10.180896    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp"]
Jul 17 21:37:10.183344 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:37:10.183319    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb28c99df_8560_4b39_9a1e_574aecf20e82.slice/crio-e18b8378bf15d72af484d87623df274a1f6a9fc0f3433c31008b3a2e75bb5641 WatchSource:0}: Error finding container e18b8378bf15d72af484d87623df274a1f6a9fc0f3433c31008b3a2e75bb5641: Status 404 returned error can't find the container with id e18b8378bf15d72af484d87623df274a1f6a9fc0f3433c31008b3a2e75bb5641
Jul 17 21:37:11.158002 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:11.157971    2569 generic.go:356] "Generic (PLEG): container finished" podID="b28c99df-8560-4b39-9a1e-574aecf20e82" containerID="d9cb648f77c37dca2b146e8cd78c86de89056db5839a94088c824ca933d5d381" exitCode=7
Jul 17 21:37:11.158414 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:11.158066    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp" event={"ID":"b28c99df-8560-4b39-9a1e-574aecf20e82","Type":"ContainerDied","Data":"d9cb648f77c37dca2b146e8cd78c86de89056db5839a94088c824ca933d5d381"}
Jul 17 21:37:11.158414 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:11.158129    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp" event={"ID":"b28c99df-8560-4b39-9a1e-574aecf20e82","Type":"ContainerStarted","Data":"e18b8378bf15d72af484d87623df274a1f6a9fc0f3433c31008b3a2e75bb5641"}
Jul 17 21:37:11.158414 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:11.158229    2569 scope.go:117] "RemoveContainer" containerID="d9cb648f77c37dca2b146e8cd78c86de89056db5839a94088c824ca933d5d381"
Jul 17 21:37:12.166727 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:12.166693    2569 generic.go:356] "Generic (PLEG): container finished" podID="b28c99df-8560-4b39-9a1e-574aecf20e82" containerID="d89b0a2a26a5132c031347e80cabf91e791b4c56ac40703214f4d81fc86cbad3" exitCode=7
Jul 17 21:37:12.167169 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:12.166761    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp" event={"ID":"b28c99df-8560-4b39-9a1e-574aecf20e82","Type":"ContainerDied","Data":"d89b0a2a26a5132c031347e80cabf91e791b4c56ac40703214f4d81fc86cbad3"}
Jul 17 21:37:12.167169 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:12.166804    2569 scope.go:117] "RemoveContainer" containerID="d9cb648f77c37dca2b146e8cd78c86de89056db5839a94088c824ca933d5d381"
Jul 17 21:37:12.167169 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:12.166992    2569 scope.go:117] "RemoveContainer" containerID="d89b0a2a26a5132c031347e80cabf91e791b4c56ac40703214f4d81fc86cbad3"
Jul 17 21:37:12.167304 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:37:12.167193    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-w46qp_odh-ai-gateway-infra(b28c99df-8560-4b39-9a1e-574aecf20e82)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp" podUID="b28c99df-8560-4b39-9a1e-574aecf20e82"
Jul 17 21:37:13.174607 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:13.174578    2569 scope.go:117] "RemoveContainer" containerID="d89b0a2a26a5132c031347e80cabf91e791b4c56ac40703214f4d81fc86cbad3"
Jul 17 21:37:13.174985 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:37:13.174786    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-w46qp_odh-ai-gateway-infra(b28c99df-8560-4b39-9a1e-574aecf20e82)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp" podUID="b28c99df-8560-4b39-9a1e-574aecf20e82"
Jul 17 21:37:24.650426 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:24.650379    2569 scope.go:117] "RemoveContainer" containerID="d89b0a2a26a5132c031347e80cabf91e791b4c56ac40703214f4d81fc86cbad3"
Jul 17 21:37:25.243917 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:25.243882    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp" event={"ID":"b28c99df-8560-4b39-9a1e-574aecf20e82","Type":"ContainerStarted","Data":"0d9088dc1f0a16f869a113c2ac30da63c50f0949741194d91b244c12b08a7f19"}
Jul 17 21:37:25.259770 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:25.259718    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp" podStartSLOduration=16.259703506 podStartE2EDuration="16.259703506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:37:25.25799339 +0000 UTC m=+1759.017560084" watchObservedRunningTime="2026-07-17 21:37:25.259703506 +0000 UTC m=+1759.019270246"
Jul 17 21:37:26.251285 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:26.251250    2569 generic.go:356] "Generic (PLEG): container finished" podID="b28c99df-8560-4b39-9a1e-574aecf20e82" containerID="0d9088dc1f0a16f869a113c2ac30da63c50f0949741194d91b244c12b08a7f19" exitCode=7
Jul 17 21:37:26.251719 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:26.251330    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp" event={"ID":"b28c99df-8560-4b39-9a1e-574aecf20e82","Type":"ContainerDied","Data":"0d9088dc1f0a16f869a113c2ac30da63c50f0949741194d91b244c12b08a7f19"}
Jul 17 21:37:26.251719 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:26.251433    2569 scope.go:117] "RemoveContainer" containerID="d89b0a2a26a5132c031347e80cabf91e791b4c56ac40703214f4d81fc86cbad3"
Jul 17 21:37:26.251719 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:26.251690    2569 scope.go:117] "RemoveContainer" containerID="0d9088dc1f0a16f869a113c2ac30da63c50f0949741194d91b244c12b08a7f19"
Jul 17 21:37:26.251950 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:37:26.251932    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-w46qp_odh-ai-gateway-infra(b28c99df-8560-4b39-9a1e-574aecf20e82)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp" podUID="b28c99df-8560-4b39-9a1e-574aecf20e82"
Jul 17 21:37:26.275559 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:26.275534    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp"]
Jul 17 21:37:27.399939 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:27.399915    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp"
Jul 17 21:37:27.420210 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:27.420178    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b28c99df-8560-4b39-9a1e-574aecf20e82-maas-api-service-ca\") pod \"b28c99df-8560-4b39-9a1e-574aecf20e82\" (UID: \"b28c99df-8560-4b39-9a1e-574aecf20e82\") "
Jul 17 21:37:27.420515 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:27.420493    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b28c99df-8560-4b39-9a1e-574aecf20e82-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b28c99df-8560-4b39-9a1e-574aecf20e82" (UID: "b28c99df-8560-4b39-9a1e-574aecf20e82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:37:27.521127 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:27.521041    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b28c99df-8560-4b39-9a1e-574aecf20e82-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:37:28.265682 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:28.265644    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp" event={"ID":"b28c99df-8560-4b39-9a1e-574aecf20e82","Type":"ContainerDied","Data":"e18b8378bf15d72af484d87623df274a1f6a9fc0f3433c31008b3a2e75bb5641"}
Jul 17 21:37:28.265682 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:28.265669    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp"
Jul 17 21:37:28.265924 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:28.265714    2569 scope.go:117] "RemoveContainer" containerID="0d9088dc1f0a16f869a113c2ac30da63c50f0949741194d91b244c12b08a7f19"
Jul 17 21:37:28.287459 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:28.287428    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp"]
Jul 17 21:37:28.293967 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:28.293941    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-w46qp"]
Jul 17 21:37:28.656508 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:28.656476    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b28c99df-8560-4b39-9a1e-574aecf20e82" path="/var/lib/kubelet/pods/b28c99df-8560-4b39-9a1e-574aecf20e82/volumes"
Jul 17 21:37:59.760608 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:59.760524    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt"]
Jul 17 21:37:59.761064 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:59.760907    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b28c99df-8560-4b39-9a1e-574aecf20e82" containerName="revoke-keys"
Jul 17 21:37:59.761064 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:59.760919    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b28c99df-8560-4b39-9a1e-574aecf20e82" containerName="revoke-keys"
Jul 17 21:37:59.761064 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:59.760933    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b28c99df-8560-4b39-9a1e-574aecf20e82" containerName="revoke-keys"
Jul 17 21:37:59.761064 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:59.760938    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b28c99df-8560-4b39-9a1e-574aecf20e82" containerName="revoke-keys"
Jul 17 21:37:59.761064 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:59.760952    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74a406f1-237a-40fc-b10e-247f5cfa34d4" containerName="revoke-keys"
Jul 17 21:37:59.761064 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:59.760957    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="74a406f1-237a-40fc-b10e-247f5cfa34d4" containerName="revoke-keys"
Jul 17 21:37:59.761064 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:59.760965    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b28c99df-8560-4b39-9a1e-574aecf20e82" containerName="revoke-keys"
Jul 17 21:37:59.761064 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:59.760971    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b28c99df-8560-4b39-9a1e-574aecf20e82" containerName="revoke-keys"
Jul 17 21:37:59.761064 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:59.761032    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="74a406f1-237a-40fc-b10e-247f5cfa34d4" containerName="revoke-keys"
Jul 17 21:37:59.761064 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:59.761040    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b28c99df-8560-4b39-9a1e-574aecf20e82" containerName="revoke-keys"
Jul 17 21:37:59.761064 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:59.761046    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b28c99df-8560-4b39-9a1e-574aecf20e82" containerName="revoke-keys"
Jul 17 21:37:59.761064 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:59.761053    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b28c99df-8560-4b39-9a1e-574aecf20e82" containerName="revoke-keys"
Jul 17 21:37:59.764089 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:59.764070    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt"
Jul 17 21:37:59.766085 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:59.766064    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:37:59.766219 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:59.766149    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:37:59.772158 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:59.772126    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt"]
Jul 17 21:37:59.902966 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:37:59.902925    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abec3200-ce3e-4224-a5ae-796038558950-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-dhsnt\" (UID: \"abec3200-ce3e-4224-a5ae-796038558950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt"
Jul 17 21:38:00.003842 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:00.003804    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abec3200-ce3e-4224-a5ae-796038558950-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-dhsnt\" (UID: \"abec3200-ce3e-4224-a5ae-796038558950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt"
Jul 17 21:38:00.004516 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:00.004494    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abec3200-ce3e-4224-a5ae-796038558950-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-dhsnt\" (UID: \"abec3200-ce3e-4224-a5ae-796038558950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt"
Jul 17 21:38:00.076365 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:00.076255    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt"
Jul 17 21:38:00.211653 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:00.211619    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt"]
Jul 17 21:38:00.211923 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:38:00.211902    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabec3200_ce3e_4224_a5ae_796038558950.slice/crio-6977c0877e265d754df276f20d2a660d866e35ffe58434ec3975f0f51c9471f8 WatchSource:0}: Error finding container 6977c0877e265d754df276f20d2a660d866e35ffe58434ec3975f0f51c9471f8: Status 404 returned error can't find the container with id 6977c0877e265d754df276f20d2a660d866e35ffe58434ec3975f0f51c9471f8
Jul 17 21:38:00.448504 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:00.448467    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt" event={"ID":"abec3200-ce3e-4224-a5ae-796038558950","Type":"ContainerStarted","Data":"6977c0877e265d754df276f20d2a660d866e35ffe58434ec3975f0f51c9471f8"}
Jul 17 21:38:01.455590 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:01.455549    2569 generic.go:356] "Generic (PLEG): container finished" podID="abec3200-ce3e-4224-a5ae-796038558950" containerID="a4250e11976838ad1dc9e14ec1e5f1c2de2dc4d4d5c355f55192dfd64f29face" exitCode=7
Jul 17 21:38:01.455983 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:01.455638    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt" event={"ID":"abec3200-ce3e-4224-a5ae-796038558950","Type":"ContainerDied","Data":"a4250e11976838ad1dc9e14ec1e5f1c2de2dc4d4d5c355f55192dfd64f29face"}
Jul 17 21:38:01.455983 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:01.455829    2569 scope.go:117] "RemoveContainer" containerID="a4250e11976838ad1dc9e14ec1e5f1c2de2dc4d4d5c355f55192dfd64f29face"
Jul 17 21:38:02.463448 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:02.463346    2569 generic.go:356] "Generic (PLEG): container finished" podID="abec3200-ce3e-4224-a5ae-796038558950" containerID="f877293e7fab7e95f1a24253766c35715b1ac05aa65d2faf8ef8eb360af5fbef" exitCode=7
Jul 17 21:38:02.463914 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:02.463450    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt" event={"ID":"abec3200-ce3e-4224-a5ae-796038558950","Type":"ContainerDied","Data":"f877293e7fab7e95f1a24253766c35715b1ac05aa65d2faf8ef8eb360af5fbef"}
Jul 17 21:38:02.463914 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:02.463520    2569 scope.go:117] "RemoveContainer" containerID="a4250e11976838ad1dc9e14ec1e5f1c2de2dc4d4d5c355f55192dfd64f29face"
Jul 17 21:38:02.463914 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:02.463697    2569 scope.go:117] "RemoveContainer" containerID="f877293e7fab7e95f1a24253766c35715b1ac05aa65d2faf8ef8eb360af5fbef"
Jul 17 21:38:02.464065 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:38:02.463922    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-dhsnt_odh-ai-gateway-infra(abec3200-ce3e-4224-a5ae-796038558950)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt" podUID="abec3200-ce3e-4224-a5ae-796038558950"
Jul 17 21:38:03.471200 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:03.471174    2569 scope.go:117] "RemoveContainer" containerID="f877293e7fab7e95f1a24253766c35715b1ac05aa65d2faf8ef8eb360af5fbef"
Jul 17 21:38:03.471656 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:38:03.471389    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-dhsnt_odh-ai-gateway-infra(abec3200-ce3e-4224-a5ae-796038558950)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt" podUID="abec3200-ce3e-4224-a5ae-796038558950"
Jul 17 21:38:06.752684 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:06.752643    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:38:06.776221 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:06.776195    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:38:08.364522 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:08.364491    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:38:17.650997 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:17.650967    2569 scope.go:117] "RemoveContainer" containerID="f877293e7fab7e95f1a24253766c35715b1ac05aa65d2faf8ef8eb360af5fbef"
Jul 17 21:38:18.557579 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:18.557497    2569 generic.go:356] "Generic (PLEG): container finished" podID="abec3200-ce3e-4224-a5ae-796038558950" containerID="eb9484e8fa055e6f22dffbf23217821f848835b67643f92bb8c3a99274089c78" exitCode=7
Jul 17 21:38:18.557579 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:18.557570    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt" event={"ID":"abec3200-ce3e-4224-a5ae-796038558950","Type":"ContainerDied","Data":"eb9484e8fa055e6f22dffbf23217821f848835b67643f92bb8c3a99274089c78"}
Jul 17 21:38:18.557770 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:18.557632    2569 scope.go:117] "RemoveContainer" containerID="f877293e7fab7e95f1a24253766c35715b1ac05aa65d2faf8ef8eb360af5fbef"
Jul 17 21:38:18.557952 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:18.557937    2569 scope.go:117] "RemoveContainer" containerID="eb9484e8fa055e6f22dffbf23217821f848835b67643f92bb8c3a99274089c78"
Jul 17 21:38:18.558160 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:38:18.558142    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-dhsnt_odh-ai-gateway-infra(abec3200-ce3e-4224-a5ae-796038558950)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt" podUID="abec3200-ce3e-4224-a5ae-796038558950"
Jul 17 21:38:19.582422 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:19.582375    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt"]
Jul 17 21:38:19.720527 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:19.720498    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt"
Jul 17 21:38:19.782950 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:19.782912    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abec3200-ce3e-4224-a5ae-796038558950-maas-api-service-ca\") pod \"abec3200-ce3e-4224-a5ae-796038558950\" (UID: \"abec3200-ce3e-4224-a5ae-796038558950\") "
Jul 17 21:38:19.783258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:19.783237    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abec3200-ce3e-4224-a5ae-796038558950-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "abec3200-ce3e-4224-a5ae-796038558950" (UID: "abec3200-ce3e-4224-a5ae-796038558950"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:38:19.884422 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:19.884324    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abec3200-ce3e-4224-a5ae-796038558950-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:38:20.571154 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:20.571124    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt"
Jul 17 21:38:20.571354 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:20.571170    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt" event={"ID":"abec3200-ce3e-4224-a5ae-796038558950","Type":"ContainerDied","Data":"6977c0877e265d754df276f20d2a660d866e35ffe58434ec3975f0f51c9471f8"}
Jul 17 21:38:20.571354 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:20.571235    2569 scope.go:117] "RemoveContainer" containerID="eb9484e8fa055e6f22dffbf23217821f848835b67643f92bb8c3a99274089c78"
Jul 17 21:38:20.594434 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:20.594388    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt"]
Jul 17 21:38:20.599563 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:20.599540    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dhsnt"]
Jul 17 21:38:20.656051 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:20.656021    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abec3200-ce3e-4224-a5ae-796038558950" path="/var/lib/kubelet/pods/abec3200-ce3e-4224-a5ae-796038558950/volumes"
Jul 17 21:38:59.794891 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:59.794847    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:38:59.817855 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:59.817828    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms"]
Jul 17 21:38:59.818253 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:59.818237    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abec3200-ce3e-4224-a5ae-796038558950" containerName="revoke-keys"
Jul 17 21:38:59.818304 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:59.818258    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="abec3200-ce3e-4224-a5ae-796038558950" containerName="revoke-keys"
Jul 17 21:38:59.818304 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:59.818269    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abec3200-ce3e-4224-a5ae-796038558950" containerName="revoke-keys"
Jul 17 21:38:59.818304 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:59.818274    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="abec3200-ce3e-4224-a5ae-796038558950" containerName="revoke-keys"
Jul 17 21:38:59.818478 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:59.818349    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="abec3200-ce3e-4224-a5ae-796038558950" containerName="revoke-keys"
Jul 17 21:38:59.818478 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:59.818357    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="abec3200-ce3e-4224-a5ae-796038558950" containerName="revoke-keys"
Jul 17 21:38:59.821382 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:59.821362    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms"
Jul 17 21:38:59.824217 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:59.824196    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:38:59.824335 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:59.824277    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:38:59.832154 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:59.832126    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms"]
Jul 17 21:38:59.890459 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:59.890419    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:38:59.951562 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:38:59.951525    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23bb5c86-4388-4862-8181-aea3deb53394-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-2lpms\" (UID: \"23bb5c86-4388-4862-8181-aea3deb53394\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms"
Jul 17 21:39:00.052256 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:00.052167    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23bb5c86-4388-4862-8181-aea3deb53394-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-2lpms\" (UID: \"23bb5c86-4388-4862-8181-aea3deb53394\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms"
Jul 17 21:39:00.052791 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:00.052772    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23bb5c86-4388-4862-8181-aea3deb53394-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-2lpms\" (UID: \"23bb5c86-4388-4862-8181-aea3deb53394\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms"
Jul 17 21:39:00.134151 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:00.134114    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms"
Jul 17 21:39:00.271660 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:00.271633    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms"]
Jul 17 21:39:00.273231 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:39:00.273205    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23bb5c86_4388_4862_8181_aea3deb53394.slice/crio-534344f6fb247ebf488c407e863eddf558cf698beaf476fe68acab91464222ab WatchSource:0}: Error finding container 534344f6fb247ebf488c407e863eddf558cf698beaf476fe68acab91464222ab: Status 404 returned error can't find the container with id 534344f6fb247ebf488c407e863eddf558cf698beaf476fe68acab91464222ab
Jul 17 21:39:00.701642 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:00.701609    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:39:00.796062 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:00.796019    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms" event={"ID":"23bb5c86-4388-4862-8181-aea3deb53394","Type":"ContainerStarted","Data":"2fd8ec15e3910aab82e0f687afaa7c6fa70867f989413978b61196baf7df52e9"}
Jul 17 21:39:00.796471 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:00.796092    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms" event={"ID":"23bb5c86-4388-4862-8181-aea3deb53394","Type":"ContainerStarted","Data":"534344f6fb247ebf488c407e863eddf558cf698beaf476fe68acab91464222ab"}
Jul 17 21:39:00.827376 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:00.827272    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms" podStartSLOduration=1.8272544499999999 podStartE2EDuration="1.82725445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:39:00.825599102 +0000 UTC m=+1854.585165797" watchObservedRunningTime="2026-07-17 21:39:00.82725445 +0000 UTC m=+1854.586821144"
Jul 17 21:39:00.827729 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:00.827686    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:39:01.803132 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:01.803088    2569 generic.go:356] "Generic (PLEG): container finished" podID="23bb5c86-4388-4862-8181-aea3deb53394" containerID="2fd8ec15e3910aab82e0f687afaa7c6fa70867f989413978b61196baf7df52e9" exitCode=7
Jul 17 21:39:01.803626 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:01.803168    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms" event={"ID":"23bb5c86-4388-4862-8181-aea3deb53394","Type":"ContainerDied","Data":"2fd8ec15e3910aab82e0f687afaa7c6fa70867f989413978b61196baf7df52e9"}
Jul 17 21:39:01.803626 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:01.803368    2569 scope.go:117] "RemoveContainer" containerID="2fd8ec15e3910aab82e0f687afaa7c6fa70867f989413978b61196baf7df52e9"
Jul 17 21:39:02.811239 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:02.811149    2569 generic.go:356] "Generic (PLEG): container finished" podID="23bb5c86-4388-4862-8181-aea3deb53394" containerID="6500c372991aa2c4a3eb63fe304b4956ff20ab4634f6a438293ed4e0a30f3355" exitCode=7
Jul 17 21:39:02.811239 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:02.811192    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms" event={"ID":"23bb5c86-4388-4862-8181-aea3deb53394","Type":"ContainerDied","Data":"6500c372991aa2c4a3eb63fe304b4956ff20ab4634f6a438293ed4e0a30f3355"}
Jul 17 21:39:02.811239 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:02.811242    2569 scope.go:117] "RemoveContainer" containerID="2fd8ec15e3910aab82e0f687afaa7c6fa70867f989413978b61196baf7df52e9"
Jul 17 21:39:02.811784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:02.811431    2569 scope.go:117] "RemoveContainer" containerID="6500c372991aa2c4a3eb63fe304b4956ff20ab4634f6a438293ed4e0a30f3355"
Jul 17 21:39:02.811784 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:39:02.811672    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-2lpms_odh-ai-gateway-infra(23bb5c86-4388-4862-8181-aea3deb53394)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms" podUID="23bb5c86-4388-4862-8181-aea3deb53394"
Jul 17 21:39:03.818598 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:03.818569    2569 scope.go:117] "RemoveContainer" containerID="6500c372991aa2c4a3eb63fe304b4956ff20ab4634f6a438293ed4e0a30f3355"
Jul 17 21:39:03.818974 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:39:03.818764    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-2lpms_odh-ai-gateway-infra(23bb5c86-4388-4862-8181-aea3deb53394)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms" podUID="23bb5c86-4388-4862-8181-aea3deb53394"
Jul 17 21:39:09.823006 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:09.822968    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:39:09.959028 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:09.958985    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:39:12.716682 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:12.716650    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:39:15.121235 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:15.121202    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:39:16.528335 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:16.528297    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:39:17.650810 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:17.650783    2569 scope.go:117] "RemoveContainer" containerID="6500c372991aa2c4a3eb63fe304b4956ff20ab4634f6a438293ed4e0a30f3355"
Jul 17 21:39:18.906712 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:18.906678    2569 generic.go:356] "Generic (PLEG): container finished" podID="23bb5c86-4388-4862-8181-aea3deb53394" containerID="ae12046b7f54bccef3b7ebcd67863a920ac5c23871cf4b67f3c7add75ac45c51" exitCode=6
Jul 17 21:39:18.907115 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:18.906742    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms" event={"ID":"23bb5c86-4388-4862-8181-aea3deb53394","Type":"ContainerDied","Data":"ae12046b7f54bccef3b7ebcd67863a920ac5c23871cf4b67f3c7add75ac45c51"}
Jul 17 21:39:18.907115 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:18.906799    2569 scope.go:117] "RemoveContainer" containerID="6500c372991aa2c4a3eb63fe304b4956ff20ab4634f6a438293ed4e0a30f3355"
Jul 17 21:39:18.907222 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:18.907203    2569 scope.go:117] "RemoveContainer" containerID="ae12046b7f54bccef3b7ebcd67863a920ac5c23871cf4b67f3c7add75ac45c51"
Jul 17 21:39:18.907463 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:39:18.907441    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-2lpms_odh-ai-gateway-infra(23bb5c86-4388-4862-8181-aea3deb53394)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms" podUID="23bb5c86-4388-4862-8181-aea3deb53394"
Jul 17 21:39:19.936708 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:19.936676    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms"]
Jul 17 21:39:20.078661 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:20.078636    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms"
Jul 17 21:39:20.126209 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:20.126180    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23bb5c86-4388-4862-8181-aea3deb53394-maas-api-service-ca\") pod \"23bb5c86-4388-4862-8181-aea3deb53394\" (UID: \"23bb5c86-4388-4862-8181-aea3deb53394\") "
Jul 17 21:39:20.126652 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:20.126626    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23bb5c86-4388-4862-8181-aea3deb53394-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "23bb5c86-4388-4862-8181-aea3deb53394" (UID: "23bb5c86-4388-4862-8181-aea3deb53394"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:39:20.227494 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:20.227421    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23bb5c86-4388-4862-8181-aea3deb53394-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:39:20.920253 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:20.920176    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms"
Jul 17 21:39:20.920433 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:20.920172    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms" event={"ID":"23bb5c86-4388-4862-8181-aea3deb53394","Type":"ContainerDied","Data":"534344f6fb247ebf488c407e863eddf558cf698beaf476fe68acab91464222ab"}
Jul 17 21:39:20.920433 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:20.920318    2569 scope.go:117] "RemoveContainer" containerID="ae12046b7f54bccef3b7ebcd67863a920ac5c23871cf4b67f3c7add75ac45c51"
Jul 17 21:39:20.939798 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:20.939771    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms"]
Jul 17 21:39:20.951997 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:20.951973    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-2lpms"]
Jul 17 21:39:22.437204 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:22.437152    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:39:22.539229 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:22.539193    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:39:22.657955 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:22.657927    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23bb5c86-4388-4862-8181-aea3deb53394" path="/var/lib/kubelet/pods/23bb5c86-4388-4862-8181-aea3deb53394/volumes"
Jul 17 21:39:22.727717 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:22.727605    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:39:22.823853 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:22.823820    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:39:22.929683 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:22.929650    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:39:24.164360 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:24.164312    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:39:24.503242 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:24.503155    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:39:24.603459 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:24.603422    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:39:24.982089 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:24.982048    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:39:25.080087 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:25.080047    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:39:28.680225 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:28.680186    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm"]
Jul 17 21:39:28.680665 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:28.680646    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23bb5c86-4388-4862-8181-aea3deb53394" containerName="revoke-keys"
Jul 17 21:39:28.680725 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:28.680669    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="23bb5c86-4388-4862-8181-aea3deb53394" containerName="revoke-keys"
Jul 17 21:39:28.680725 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:28.680713    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abec3200-ce3e-4224-a5ae-796038558950" containerName="revoke-keys"
Jul 17 21:39:28.680725 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:28.680723    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="abec3200-ce3e-4224-a5ae-796038558950" containerName="revoke-keys"
Jul 17 21:39:28.680834 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:28.680734    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23bb5c86-4388-4862-8181-aea3deb53394" containerName="revoke-keys"
Jul 17 21:39:28.680834 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:28.680743    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="23bb5c86-4388-4862-8181-aea3deb53394" containerName="revoke-keys"
Jul 17 21:39:28.681067 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:28.680843    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="23bb5c86-4388-4862-8181-aea3deb53394" containerName="revoke-keys"
Jul 17 21:39:28.681067 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:28.680856    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="23bb5c86-4388-4862-8181-aea3deb53394" containerName="revoke-keys"
Jul 17 21:39:28.681067 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:28.680864    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="abec3200-ce3e-4224-a5ae-796038558950" containerName="revoke-keys"
Jul 17 21:39:28.685174 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:28.685159    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm"
Jul 17 21:39:28.687696 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:28.687675    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:39:28.688098 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:28.688080    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:39:28.694421 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:28.694385    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm"]
Jul 17 21:39:28.803341 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:28.803306    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6579aaa3-ae44-4b10-9dcf-3d45d23960da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm\" (UID: \"6579aaa3-ae44-4b10-9dcf-3d45d23960da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm"
Jul 17 21:39:28.904943 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:28.904895    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6579aaa3-ae44-4b10-9dcf-3d45d23960da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm\" (UID: \"6579aaa3-ae44-4b10-9dcf-3d45d23960da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm"
Jul 17 21:39:28.905764 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:28.905734    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6579aaa3-ae44-4b10-9dcf-3d45d23960da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm\" (UID: \"6579aaa3-ae44-4b10-9dcf-3d45d23960da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm"
Jul 17 21:39:28.996548 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:28.996465    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm"
Jul 17 21:39:29.138604 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:29.138581    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm"]
Jul 17 21:39:29.140680 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:39:29.140649    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6579aaa3_ae44_4b10_9dcf_3d45d23960da.slice/crio-1624b5309c61d7633fb6598d71612bd16a25a40f393d8b2f80b20e4bac418b68 WatchSource:0}: Error finding container 1624b5309c61d7633fb6598d71612bd16a25a40f393d8b2f80b20e4bac418b68: Status 404 returned error can't find the container with id 1624b5309c61d7633fb6598d71612bd16a25a40f393d8b2f80b20e4bac418b68
Jul 17 21:39:29.980273 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:29.980226    2569 generic.go:356] "Generic (PLEG): container finished" podID="6579aaa3-ae44-4b10-9dcf-3d45d23960da" containerID="5a27fb5c8f2f580b29b8b4fdeb45aec5d81becde5795967947e1e87e5ca90dc4" exitCode=7
Jul 17 21:39:29.980763 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:29.980327    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm" event={"ID":"6579aaa3-ae44-4b10-9dcf-3d45d23960da","Type":"ContainerDied","Data":"5a27fb5c8f2f580b29b8b4fdeb45aec5d81becde5795967947e1e87e5ca90dc4"}
Jul 17 21:39:29.980763 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:29.980393    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm" event={"ID":"6579aaa3-ae44-4b10-9dcf-3d45d23960da","Type":"ContainerStarted","Data":"1624b5309c61d7633fb6598d71612bd16a25a40f393d8b2f80b20e4bac418b68"}
Jul 17 21:39:29.980763 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:29.980579    2569 scope.go:117] "RemoveContainer" containerID="5a27fb5c8f2f580b29b8b4fdeb45aec5d81becde5795967947e1e87e5ca90dc4"
Jul 17 21:39:30.988077 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:30.988043    2569 generic.go:356] "Generic (PLEG): container finished" podID="6579aaa3-ae44-4b10-9dcf-3d45d23960da" containerID="a32768545f906cb84c8829e549fe32531b234647ba9f0a99c9408d2cb52e713e" exitCode=7
Jul 17 21:39:30.988590 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:30.988126    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm" event={"ID":"6579aaa3-ae44-4b10-9dcf-3d45d23960da","Type":"ContainerDied","Data":"a32768545f906cb84c8829e549fe32531b234647ba9f0a99c9408d2cb52e713e"}
Jul 17 21:39:30.988590 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:30.988188    2569 scope.go:117] "RemoveContainer" containerID="5a27fb5c8f2f580b29b8b4fdeb45aec5d81becde5795967947e1e87e5ca90dc4"
Jul 17 21:39:30.988590 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:30.988438    2569 scope.go:117] "RemoveContainer" containerID="a32768545f906cb84c8829e549fe32531b234647ba9f0a99c9408d2cb52e713e"
Jul 17 21:39:30.988764 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:39:30.988698    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm_odh-ai-gateway-infra(6579aaa3-ae44-4b10-9dcf-3d45d23960da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm" podUID="6579aaa3-ae44-4b10-9dcf-3d45d23960da"
Jul 17 21:39:31.995633 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:31.995605    2569 scope.go:117] "RemoveContainer" containerID="a32768545f906cb84c8829e549fe32531b234647ba9f0a99c9408d2cb52e713e"
Jul 17 21:39:31.996039 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:39:31.995799    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm_odh-ai-gateway-infra(6579aaa3-ae44-4b10-9dcf-3d45d23960da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm" podUID="6579aaa3-ae44-4b10-9dcf-3d45d23960da"
Jul 17 21:39:32.118929 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:32.118892    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:39:44.650535 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:44.650501    2569 scope.go:117] "RemoveContainer" containerID="a32768545f906cb84c8829e549fe32531b234647ba9f0a99c9408d2cb52e713e"
Jul 17 21:39:45.070976 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:45.070943    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm" event={"ID":"6579aaa3-ae44-4b10-9dcf-3d45d23960da","Type":"ContainerStarted","Data":"72ec4bfae023a34e773373c0aa9e8775fc8f6e0bb8f24f9f84e0b9a515ace9fa"}
Jul 17 21:39:45.088902 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:45.088849    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm" podStartSLOduration=17.088834667 podStartE2EDuration="17.088834667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:39:45.086650727 +0000 UTC m=+1898.846217431" watchObservedRunningTime="2026-07-17 21:39:45.088834667 +0000 UTC m=+1898.848401405"
Jul 17 21:39:46.078231 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:46.078201    2569 generic.go:356] "Generic (PLEG): container finished" podID="6579aaa3-ae44-4b10-9dcf-3d45d23960da" containerID="72ec4bfae023a34e773373c0aa9e8775fc8f6e0bb8f24f9f84e0b9a515ace9fa" exitCode=7
Jul 17 21:39:46.078633 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:46.078276    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm" event={"ID":"6579aaa3-ae44-4b10-9dcf-3d45d23960da","Type":"ContainerDied","Data":"72ec4bfae023a34e773373c0aa9e8775fc8f6e0bb8f24f9f84e0b9a515ace9fa"}
Jul 17 21:39:46.078633 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:46.078334    2569 scope.go:117] "RemoveContainer" containerID="a32768545f906cb84c8829e549fe32531b234647ba9f0a99c9408d2cb52e713e"
Jul 17 21:39:46.078724 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:46.078645    2569 scope.go:117] "RemoveContainer" containerID="72ec4bfae023a34e773373c0aa9e8775fc8f6e0bb8f24f9f84e0b9a515ace9fa"
Jul 17 21:39:46.078842 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:39:46.078824    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm_odh-ai-gateway-infra(6579aaa3-ae44-4b10-9dcf-3d45d23960da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm" podUID="6579aaa3-ae44-4b10-9dcf-3d45d23960da"
Jul 17 21:39:46.106103 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:46.106080    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm"]
Jul 17 21:39:46.210701 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:46.210648    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:39:46.285799 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:46.285764    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:39:47.228333 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:47.228310    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm"
Jul 17 21:39:47.372554 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:47.372468    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6579aaa3-ae44-4b10-9dcf-3d45d23960da-maas-api-service-ca\") pod \"6579aaa3-ae44-4b10-9dcf-3d45d23960da\" (UID: \"6579aaa3-ae44-4b10-9dcf-3d45d23960da\") "
Jul 17 21:39:47.372866 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:47.372841    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6579aaa3-ae44-4b10-9dcf-3d45d23960da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6579aaa3-ae44-4b10-9dcf-3d45d23960da" (UID: "6579aaa3-ae44-4b10-9dcf-3d45d23960da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:39:47.473749 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:47.473720    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6579aaa3-ae44-4b10-9dcf-3d45d23960da-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:39:48.091607 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:48.091578    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm"
Jul 17 21:39:48.091851 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:48.091622    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm" event={"ID":"6579aaa3-ae44-4b10-9dcf-3d45d23960da","Type":"ContainerDied","Data":"1624b5309c61d7633fb6598d71612bd16a25a40f393d8b2f80b20e4bac418b68"}
Jul 17 21:39:48.091851 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:48.091698    2569 scope.go:117] "RemoveContainer" containerID="72ec4bfae023a34e773373c0aa9e8775fc8f6e0bb8f24f9f84e0b9a515ace9fa"
Jul 17 21:39:48.114742 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:48.114718    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm"]
Jul 17 21:39:48.121010 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:48.120988    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-przbm"]
Jul 17 21:39:48.655924 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:48.655889    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6579aaa3-ae44-4b10-9dcf-3d45d23960da" path="/var/lib/kubelet/pods/6579aaa3-ae44-4b10-9dcf-3d45d23960da/volumes"
Jul 17 21:39:49.673789 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:49.673760    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:39:59.890247 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:59.890211    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p"]
Jul 17 21:39:59.890713 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:59.890694    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6579aaa3-ae44-4b10-9dcf-3d45d23960da" containerName="revoke-keys"
Jul 17 21:39:59.890758 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:59.890715    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6579aaa3-ae44-4b10-9dcf-3d45d23960da" containerName="revoke-keys"
Jul 17 21:39:59.890758 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:59.890723    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6579aaa3-ae44-4b10-9dcf-3d45d23960da" containerName="revoke-keys"
Jul 17 21:39:59.890758 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:59.890728    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6579aaa3-ae44-4b10-9dcf-3d45d23960da" containerName="revoke-keys"
Jul 17 21:39:59.890758 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:59.890742    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23bb5c86-4388-4862-8181-aea3deb53394" containerName="revoke-keys"
Jul 17 21:39:59.890758 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:59.890748    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="23bb5c86-4388-4862-8181-aea3deb53394" containerName="revoke-keys"
Jul 17 21:39:59.890909 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:59.890803    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6579aaa3-ae44-4b10-9dcf-3d45d23960da" containerName="revoke-keys"
Jul 17 21:39:59.890909 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:59.890810    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="23bb5c86-4388-4862-8181-aea3deb53394" containerName="revoke-keys"
Jul 17 21:39:59.890909 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:59.890816    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6579aaa3-ae44-4b10-9dcf-3d45d23960da" containerName="revoke-keys"
Jul 17 21:39:59.890909 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:59.890821    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6579aaa3-ae44-4b10-9dcf-3d45d23960da" containerName="revoke-keys"
Jul 17 21:39:59.895360 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:59.895341    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p"
Jul 17 21:39:59.897611 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:59.897592    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:39:59.897705 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:59.897637    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:39:59.901800 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:59.901776    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p"]
Jul 17 21:39:59.984109 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:39:59.984060    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b557d97-24b3-4a68-99c1-02bf772148d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-lc64p\" (UID: \"8b557d97-24b3-4a68-99c1-02bf772148d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p"
Jul 17 21:40:00.085089 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:00.085055    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b557d97-24b3-4a68-99c1-02bf772148d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-lc64p\" (UID: \"8b557d97-24b3-4a68-99c1-02bf772148d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p"
Jul 17 21:40:00.085733 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:00.085707    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b557d97-24b3-4a68-99c1-02bf772148d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-lc64p\" (UID: \"8b557d97-24b3-4a68-99c1-02bf772148d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p"
Jul 17 21:40:00.206804 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:00.206732    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p"
Jul 17 21:40:00.338464 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:00.338433    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p"]
Jul 17 21:40:00.340024 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:40:00.339993    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b557d97_24b3_4a68_99c1_02bf772148d5.slice/crio-46d9d67f90183df8b1061f970b9eeff295878901713b9d0edbcef0ef986567b8 WatchSource:0}: Error finding container 46d9d67f90183df8b1061f970b9eeff295878901713b9d0edbcef0ef986567b8: Status 404 returned error can't find the container with id 46d9d67f90183df8b1061f970b9eeff295878901713b9d0edbcef0ef986567b8
Jul 17 21:40:01.164825 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:01.164788    2569 generic.go:356] "Generic (PLEG): container finished" podID="8b557d97-24b3-4a68-99c1-02bf772148d5" containerID="e05d620ba5b4da01df4c301873d6d9a4eb2e50ff7309ba0d91110568287fc2f1" exitCode=6
Jul 17 21:40:01.165213 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:01.164872    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p" event={"ID":"8b557d97-24b3-4a68-99c1-02bf772148d5","Type":"ContainerDied","Data":"e05d620ba5b4da01df4c301873d6d9a4eb2e50ff7309ba0d91110568287fc2f1"}
Jul 17 21:40:01.165213 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:01.164936    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p" event={"ID":"8b557d97-24b3-4a68-99c1-02bf772148d5","Type":"ContainerStarted","Data":"46d9d67f90183df8b1061f970b9eeff295878901713b9d0edbcef0ef986567b8"}
Jul 17 21:40:01.165213 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:01.165032    2569 scope.go:117] "RemoveContainer" containerID="e05d620ba5b4da01df4c301873d6d9a4eb2e50ff7309ba0d91110568287fc2f1"
Jul 17 21:40:02.172657 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:02.172619    2569 generic.go:356] "Generic (PLEG): container finished" podID="8b557d97-24b3-4a68-99c1-02bf772148d5" containerID="e0d967bea12635f50020adec2e077e0466e0171d39a55d98190e84740ff81c3b" exitCode=6
Jul 17 21:40:02.173120 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:02.172689    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p" event={"ID":"8b557d97-24b3-4a68-99c1-02bf772148d5","Type":"ContainerDied","Data":"e0d967bea12635f50020adec2e077e0466e0171d39a55d98190e84740ff81c3b"}
Jul 17 21:40:02.173120 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:02.172751    2569 scope.go:117] "RemoveContainer" containerID="e05d620ba5b4da01df4c301873d6d9a4eb2e50ff7309ba0d91110568287fc2f1"
Jul 17 21:40:02.173120 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:02.172939    2569 scope.go:117] "RemoveContainer" containerID="e0d967bea12635f50020adec2e077e0466e0171d39a55d98190e84740ff81c3b"
Jul 17 21:40:02.173261 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:40:02.173170    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-lc64p_odh-ai-gateway-infra(8b557d97-24b3-4a68-99c1-02bf772148d5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p" podUID="8b557d97-24b3-4a68-99c1-02bf772148d5"
Jul 17 21:40:03.181190 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:03.181162    2569 scope.go:117] "RemoveContainer" containerID="e0d967bea12635f50020adec2e077e0466e0171d39a55d98190e84740ff81c3b"
Jul 17 21:40:03.181598 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:40:03.181350    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-lc64p_odh-ai-gateway-infra(8b557d97-24b3-4a68-99c1-02bf772148d5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p" podUID="8b557d97-24b3-4a68-99c1-02bf772148d5"
Jul 17 21:40:15.650721 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:15.650684    2569 scope.go:117] "RemoveContainer" containerID="e0d967bea12635f50020adec2e077e0466e0171d39a55d98190e84740ff81c3b"
Jul 17 21:40:16.256022 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:16.255985    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p" event={"ID":"8b557d97-24b3-4a68-99c1-02bf772148d5","Type":"ContainerStarted","Data":"d3e53a27804516bfe0ec90a6ae5e7a91f27aeecb2fb0b2f29f3555c964b8637f"}
Jul 17 21:40:16.273049 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:16.273001    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p" podStartSLOduration=17.272985586 podStartE2EDuration="17.272985586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:40:16.271505465 +0000 UTC m=+1930.031072158" watchObservedRunningTime="2026-07-17 21:40:16.272985586 +0000 UTC m=+1930.032552279"
Jul 17 21:40:17.262891 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:17.262856    2569 generic.go:356] "Generic (PLEG): container finished" podID="8b557d97-24b3-4a68-99c1-02bf772148d5" containerID="d3e53a27804516bfe0ec90a6ae5e7a91f27aeecb2fb0b2f29f3555c964b8637f" exitCode=6
Jul 17 21:40:17.262891 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:17.262894    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p" event={"ID":"8b557d97-24b3-4a68-99c1-02bf772148d5","Type":"ContainerDied","Data":"d3e53a27804516bfe0ec90a6ae5e7a91f27aeecb2fb0b2f29f3555c964b8637f"}
Jul 17 21:40:17.263375 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:17.262963    2569 scope.go:117] "RemoveContainer" containerID="e0d967bea12635f50020adec2e077e0466e0171d39a55d98190e84740ff81c3b"
Jul 17 21:40:17.263375 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:17.263333    2569 scope.go:117] "RemoveContainer" containerID="d3e53a27804516bfe0ec90a6ae5e7a91f27aeecb2fb0b2f29f3555c964b8637f"
Jul 17 21:40:17.263607 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:40:17.263582    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-lc64p_odh-ai-gateway-infra(8b557d97-24b3-4a68-99c1-02bf772148d5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p" podUID="8b557d97-24b3-4a68-99c1-02bf772148d5"
Jul 17 21:40:17.283436 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:17.283409    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p"]
Jul 17 21:40:18.431117 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:18.431088    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p"
Jul 17 21:40:18.553670 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:18.553585    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b557d97-24b3-4a68-99c1-02bf772148d5-maas-api-service-ca\") pod \"8b557d97-24b3-4a68-99c1-02bf772148d5\" (UID: \"8b557d97-24b3-4a68-99c1-02bf772148d5\") "
Jul 17 21:40:18.553945 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:18.553922    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b557d97-24b3-4a68-99c1-02bf772148d5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8b557d97-24b3-4a68-99c1-02bf772148d5" (UID: "8b557d97-24b3-4a68-99c1-02bf772148d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:40:18.654393 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:18.654358    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b557d97-24b3-4a68-99c1-02bf772148d5-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:40:18.723970 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:18.723938    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z"]
Jul 17 21:40:18.724328 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:18.724316    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b557d97-24b3-4a68-99c1-02bf772148d5" containerName="revoke-keys"
Jul 17 21:40:18.724371 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:18.724330    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b557d97-24b3-4a68-99c1-02bf772148d5" containerName="revoke-keys"
Jul 17 21:40:18.724371 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:18.724339    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b557d97-24b3-4a68-99c1-02bf772148d5" containerName="revoke-keys"
Jul 17 21:40:18.724371 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:18.724345    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b557d97-24b3-4a68-99c1-02bf772148d5" containerName="revoke-keys"
Jul 17 21:40:18.724371 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:18.724367    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6579aaa3-ae44-4b10-9dcf-3d45d23960da" containerName="revoke-keys"
Jul 17 21:40:18.724538 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:18.724373    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6579aaa3-ae44-4b10-9dcf-3d45d23960da" containerName="revoke-keys"
Jul 17 21:40:18.724538 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:18.724466    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b557d97-24b3-4a68-99c1-02bf772148d5" containerName="revoke-keys"
Jul 17 21:40:18.724538 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:18.724474    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b557d97-24b3-4a68-99c1-02bf772148d5" containerName="revoke-keys"
Jul 17 21:40:18.724538 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:18.724481    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b557d97-24b3-4a68-99c1-02bf772148d5" containerName="revoke-keys"
Jul 17 21:40:18.727560 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:18.727542    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z"
Jul 17 21:40:18.735477 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:18.735454    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z"]
Jul 17 21:40:18.855295 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:18.855263    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e741fa-ea82-4432-9266-9068289183ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z\" (UID: \"42e741fa-ea82-4432-9266-9068289183ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z"
Jul 17 21:40:18.956129 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:18.956096    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e741fa-ea82-4432-9266-9068289183ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z\" (UID: \"42e741fa-ea82-4432-9266-9068289183ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z"
Jul 17 21:40:18.956761 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:18.956740    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e741fa-ea82-4432-9266-9068289183ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z\" (UID: \"42e741fa-ea82-4432-9266-9068289183ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z"
Jul 17 21:40:19.039508 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:19.039462    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z"
Jul 17 21:40:19.178761 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:19.178737    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z"]
Jul 17 21:40:19.180784 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:40:19.180756    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e741fa_ea82_4432_9266_9068289183ae.slice/crio-4c5ee4ea6022505f633c167e2244fce814911f9a959af61c053a3ee1e94f564c WatchSource:0}: Error finding container 4c5ee4ea6022505f633c167e2244fce814911f9a959af61c053a3ee1e94f564c: Status 404 returned error can't find the container with id 4c5ee4ea6022505f633c167e2244fce814911f9a959af61c053a3ee1e94f564c
Jul 17 21:40:19.275996 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:19.275966    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z" event={"ID":"42e741fa-ea82-4432-9266-9068289183ae","Type":"ContainerStarted","Data":"4c5ee4ea6022505f633c167e2244fce814911f9a959af61c053a3ee1e94f564c"}
Jul 17 21:40:19.277230 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:19.277207    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p" event={"ID":"8b557d97-24b3-4a68-99c1-02bf772148d5","Type":"ContainerDied","Data":"46d9d67f90183df8b1061f970b9eeff295878901713b9d0edbcef0ef986567b8"}
Jul 17 21:40:19.277342 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:19.277253    2569 scope.go:117] "RemoveContainer" containerID="d3e53a27804516bfe0ec90a6ae5e7a91f27aeecb2fb0b2f29f3555c964b8637f"
Jul 17 21:40:19.277342 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:19.277214    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p"
Jul 17 21:40:19.292460 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:19.292436    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p"]
Jul 17 21:40:19.297484 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:19.297463    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-lc64p"]
Jul 17 21:40:20.284702 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:20.284667    2569 generic.go:356] "Generic (PLEG): container finished" podID="42e741fa-ea82-4432-9266-9068289183ae" containerID="e36e8f93e5577470846b9f3473733f861012b6268d8e67a620087fbc10257fca" exitCode=7
Jul 17 21:40:20.285115 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:20.284727    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z" event={"ID":"42e741fa-ea82-4432-9266-9068289183ae","Type":"ContainerDied","Data":"e36e8f93e5577470846b9f3473733f861012b6268d8e67a620087fbc10257fca"}
Jul 17 21:40:20.285115 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:20.284901    2569 scope.go:117] "RemoveContainer" containerID="e36e8f93e5577470846b9f3473733f861012b6268d8e67a620087fbc10257fca"
Jul 17 21:40:20.657197 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:20.657162    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b557d97-24b3-4a68-99c1-02bf772148d5" path="/var/lib/kubelet/pods/8b557d97-24b3-4a68-99c1-02bf772148d5/volumes"
Jul 17 21:40:21.291817 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:21.291723    2569 generic.go:356] "Generic (PLEG): container finished" podID="42e741fa-ea82-4432-9266-9068289183ae" containerID="afcc4b513150ab84c423417417587a1a7ce5d0fba62270919d19fa627b6eb389" exitCode=7
Jul 17 21:40:21.291817 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:21.291774    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z" event={"ID":"42e741fa-ea82-4432-9266-9068289183ae","Type":"ContainerDied","Data":"afcc4b513150ab84c423417417587a1a7ce5d0fba62270919d19fa627b6eb389"}
Jul 17 21:40:21.292279 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:21.291847    2569 scope.go:117] "RemoveContainer" containerID="e36e8f93e5577470846b9f3473733f861012b6268d8e67a620087fbc10257fca"
Jul 17 21:40:21.292279 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:21.292104    2569 scope.go:117] "RemoveContainer" containerID="afcc4b513150ab84c423417417587a1a7ce5d0fba62270919d19fa627b6eb389"
Jul 17 21:40:21.292381 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:40:21.292312    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z_odh-ai-gateway-infra(42e741fa-ea82-4432-9266-9068289183ae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z" podUID="42e741fa-ea82-4432-9266-9068289183ae"
Jul 17 21:40:22.298796 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:22.298768    2569 scope.go:117] "RemoveContainer" containerID="afcc4b513150ab84c423417417587a1a7ce5d0fba62270919d19fa627b6eb389"
Jul 17 21:40:22.299186 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:40:22.298948    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z_odh-ai-gateway-infra(42e741fa-ea82-4432-9266-9068289183ae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z" podUID="42e741fa-ea82-4432-9266-9068289183ae"
Jul 17 21:40:25.933747 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:25.933700    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:40:25.989841 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:25.989807    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:40:28.820101 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:28.820063    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:40:28.905073 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:28.905035    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:40:31.636222 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:31.636184    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:40:33.964786 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:33.964737    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:40:34.878187 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:34.878153    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:40:35.650148 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:35.650116    2569 scope.go:117] "RemoveContainer" containerID="afcc4b513150ab84c423417417587a1a7ce5d0fba62270919d19fa627b6eb389"
Jul 17 21:40:35.914567 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:35.914480    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:40:36.381022 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:36.380990    2569 generic.go:356] "Generic (PLEG): container finished" podID="42e741fa-ea82-4432-9266-9068289183ae" containerID="66a49110972979f0d59117f922ebb8b4610021b5664f8a799b649311cd620684" exitCode=6
Jul 17 21:40:36.381213 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:36.381061    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z" event={"ID":"42e741fa-ea82-4432-9266-9068289183ae","Type":"ContainerDied","Data":"66a49110972979f0d59117f922ebb8b4610021b5664f8a799b649311cd620684"}
Jul 17 21:40:36.381213 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:36.381121    2569 scope.go:117] "RemoveContainer" containerID="afcc4b513150ab84c423417417587a1a7ce5d0fba62270919d19fa627b6eb389"
Jul 17 21:40:36.381487 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:36.381471    2569 scope.go:117] "RemoveContainer" containerID="66a49110972979f0d59117f922ebb8b4610021b5664f8a799b649311cd620684"
Jul 17 21:40:36.381726 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:40:36.381708    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z_odh-ai-gateway-infra(42e741fa-ea82-4432-9266-9068289183ae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z" podUID="42e741fa-ea82-4432-9266-9068289183ae"
Jul 17 21:40:37.408130 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:37.408093    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z"]
Jul 17 21:40:37.547023 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:37.546994    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z"
Jul 17 21:40:37.722376 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:37.722279    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e741fa-ea82-4432-9266-9068289183ae-maas-api-service-ca\") pod \"42e741fa-ea82-4432-9266-9068289183ae\" (UID: \"42e741fa-ea82-4432-9266-9068289183ae\") "
Jul 17 21:40:37.722686 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:37.722662    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42e741fa-ea82-4432-9266-9068289183ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42e741fa-ea82-4432-9266-9068289183ae" (UID: "42e741fa-ea82-4432-9266-9068289183ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:40:37.822924 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:37.822890    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e741fa-ea82-4432-9266-9068289183ae-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:40:38.395025 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:38.394993    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z"
Jul 17 21:40:38.395202 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:38.395028    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z" event={"ID":"42e741fa-ea82-4432-9266-9068289183ae","Type":"ContainerDied","Data":"4c5ee4ea6022505f633c167e2244fce814911f9a959af61c053a3ee1e94f564c"}
Jul 17 21:40:38.395202 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:38.395083    2569 scope.go:117] "RemoveContainer" containerID="66a49110972979f0d59117f922ebb8b4610021b5664f8a799b649311cd620684"
Jul 17 21:40:38.417185 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:38.417157    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z"]
Jul 17 21:40:38.422024 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:38.421997    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wrv7z"]
Jul 17 21:40:38.657115 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:38.657027    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42e741fa-ea82-4432-9266-9068289183ae" path="/var/lib/kubelet/pods/42e741fa-ea82-4432-9266-9068289183ae/volumes"
Jul 17 21:40:40.460753 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:40.460715    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:40:40.562258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:40.562221    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:40:40.701888 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:40.701851    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:40:40.797453 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:40.797360    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:40:41.878488 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:41.878449    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:40:41.962843 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:41.962807    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:40:42.029977 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:42.029936    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:40:42.427862 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:42.427830    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:40:42.574954 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:42.574912    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:40:42.685153 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:42.685038    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:40:44.185659 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:44.185622    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:40:44.824722 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:44.824685    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:40:46.824543 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:46.824508    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p"]
Jul 17 21:40:46.824925 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:46.824887    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e741fa-ea82-4432-9266-9068289183ae" containerName="revoke-keys"
Jul 17 21:40:46.824925 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:46.824899    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e741fa-ea82-4432-9266-9068289183ae" containerName="revoke-keys"
Jul 17 21:40:46.824925 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:46.824914    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e741fa-ea82-4432-9266-9068289183ae" containerName="revoke-keys"
Jul 17 21:40:46.824925 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:46.824919    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e741fa-ea82-4432-9266-9068289183ae" containerName="revoke-keys"
Jul 17 21:40:46.824925 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:46.824927    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b557d97-24b3-4a68-99c1-02bf772148d5" containerName="revoke-keys"
Jul 17 21:40:46.825084 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:46.824933    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b557d97-24b3-4a68-99c1-02bf772148d5" containerName="revoke-keys"
Jul 17 21:40:46.825084 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:46.824999    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e741fa-ea82-4432-9266-9068289183ae" containerName="revoke-keys"
Jul 17 21:40:46.825084 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:46.825012    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e741fa-ea82-4432-9266-9068289183ae" containerName="revoke-keys"
Jul 17 21:40:46.828066 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:46.828048    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p"
Jul 17 21:40:46.830243 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:46.830217    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:40:46.830243 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:46.830234    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:40:46.835084 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:46.835059    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p"]
Jul 17 21:40:46.905054 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:46.905015    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5a4983-99c9-49bd-a96c-343ce782192e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p\" (UID: \"bb5a4983-99c9-49bd-a96c-343ce782192e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p"
Jul 17 21:40:47.005832 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:47.005792    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5a4983-99c9-49bd-a96c-343ce782192e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p\" (UID: \"bb5a4983-99c9-49bd-a96c-343ce782192e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p"
Jul 17 21:40:47.006543 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:47.006522    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5a4983-99c9-49bd-a96c-343ce782192e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p\" (UID: \"bb5a4983-99c9-49bd-a96c-343ce782192e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p"
Jul 17 21:40:47.140313 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:47.140275    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p"
Jul 17 21:40:47.284431 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:47.284381    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p"]
Jul 17 21:40:47.448523 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:47.448438    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p" event={"ID":"bb5a4983-99c9-49bd-a96c-343ce782192e","Type":"ContainerStarted","Data":"b415486a9a9fde92d98160a0b6cf2f99f1205b80117e9cb2e285d46fc800c15f"}
Jul 17 21:40:48.454944 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:48.454905    2569 generic.go:356] "Generic (PLEG): container finished" podID="bb5a4983-99c9-49bd-a96c-343ce782192e" containerID="5f2c00c40b9d3b8105d4b1c80b1149cd519c563dfb6685b7b2f3422325652424" exitCode=7
Jul 17 21:40:48.455440 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:48.454961    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p" event={"ID":"bb5a4983-99c9-49bd-a96c-343ce782192e","Type":"ContainerDied","Data":"5f2c00c40b9d3b8105d4b1c80b1149cd519c563dfb6685b7b2f3422325652424"}
Jul 17 21:40:48.455440 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:48.455206    2569 scope.go:117] "RemoveContainer" containerID="5f2c00c40b9d3b8105d4b1c80b1149cd519c563dfb6685b7b2f3422325652424"
Jul 17 21:40:49.463745 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:49.463655    2569 generic.go:356] "Generic (PLEG): container finished" podID="bb5a4983-99c9-49bd-a96c-343ce782192e" containerID="a3e60df4606ba8b19abff8be83b9e264fcde330c840f784033547a6aff74a67c" exitCode=7
Jul 17 21:40:49.464196 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:49.463742    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p" event={"ID":"bb5a4983-99c9-49bd-a96c-343ce782192e","Type":"ContainerDied","Data":"a3e60df4606ba8b19abff8be83b9e264fcde330c840f784033547a6aff74a67c"}
Jul 17 21:40:49.464196 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:49.463812    2569 scope.go:117] "RemoveContainer" containerID="5f2c00c40b9d3b8105d4b1c80b1149cd519c563dfb6685b7b2f3422325652424"
Jul 17 21:40:49.464196 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:49.463922    2569 scope.go:117] "RemoveContainer" containerID="a3e60df4606ba8b19abff8be83b9e264fcde330c840f784033547a6aff74a67c"
Jul 17 21:40:49.464196 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:40:49.464174    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p_odh-ai-gateway-infra(bb5a4983-99c9-49bd-a96c-343ce782192e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p" podUID="bb5a4983-99c9-49bd-a96c-343ce782192e"
Jul 17 21:40:49.945124 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:49.945085    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl"]
Jul 17 21:40:49.945524 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:49.945511    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e741fa-ea82-4432-9266-9068289183ae" containerName="revoke-keys"
Jul 17 21:40:49.945581 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:49.945526    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e741fa-ea82-4432-9266-9068289183ae" containerName="revoke-keys"
Jul 17 21:40:49.945617 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:49.945586    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e741fa-ea82-4432-9266-9068289183ae" containerName="revoke-keys"
Jul 17 21:40:49.949912 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:49.949878    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl"
Jul 17 21:40:49.957638 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:49.957604    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl"]
Jul 17 21:40:50.027376 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:50.027331    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a07b1d-71c6-46ef-bdd9-ff380101bbc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-v5hjl\" (UID: \"22a07b1d-71c6-46ef-bdd9-ff380101bbc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl"
Jul 17 21:40:50.128279 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:50.128247    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a07b1d-71c6-46ef-bdd9-ff380101bbc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-v5hjl\" (UID: \"22a07b1d-71c6-46ef-bdd9-ff380101bbc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl"
Jul 17 21:40:50.128977 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:50.128955    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a07b1d-71c6-46ef-bdd9-ff380101bbc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-v5hjl\" (UID: \"22a07b1d-71c6-46ef-bdd9-ff380101bbc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl"
Jul 17 21:40:50.262667 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:50.262572    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl"
Jul 17 21:40:50.412795 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:50.412768    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl"]
Jul 17 21:40:50.414868 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:40:50.414843    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a07b1d_71c6_46ef_bdd9_ff380101bbc2.slice/crio-1a523d428e34edfabac416c7c2084fb2173a3611949a1eaa35c542946f028e47 WatchSource:0}: Error finding container 1a523d428e34edfabac416c7c2084fb2173a3611949a1eaa35c542946f028e47: Status 404 returned error can't find the container with id 1a523d428e34edfabac416c7c2084fb2173a3611949a1eaa35c542946f028e47
Jul 17 21:40:50.471989 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:50.471961    2569 scope.go:117] "RemoveContainer" containerID="a3e60df4606ba8b19abff8be83b9e264fcde330c840f784033547a6aff74a67c"
Jul 17 21:40:50.472459 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:40:50.472197    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p_odh-ai-gateway-infra(bb5a4983-99c9-49bd-a96c-343ce782192e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p" podUID="bb5a4983-99c9-49bd-a96c-343ce782192e"
Jul 17 21:40:50.473003 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:50.472983    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl" event={"ID":"22a07b1d-71c6-46ef-bdd9-ff380101bbc2","Type":"ContainerStarted","Data":"1a523d428e34edfabac416c7c2084fb2173a3611949a1eaa35c542946f028e47"}
Jul 17 21:40:51.479979 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:51.479945    2569 generic.go:356] "Generic (PLEG): container finished" podID="22a07b1d-71c6-46ef-bdd9-ff380101bbc2" containerID="3636dd2f890ec03d51beb4c8229f7cae051e5e889767c29ed35832e7fb1a2c2b" exitCode=6
Jul 17 21:40:51.480333 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:51.480028    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl" event={"ID":"22a07b1d-71c6-46ef-bdd9-ff380101bbc2","Type":"ContainerDied","Data":"3636dd2f890ec03d51beb4c8229f7cae051e5e889767c29ed35832e7fb1a2c2b"}
Jul 17 21:40:51.480333 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:51.480243    2569 scope.go:117] "RemoveContainer" containerID="3636dd2f890ec03d51beb4c8229f7cae051e5e889767c29ed35832e7fb1a2c2b"
Jul 17 21:40:52.487744 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:52.487655    2569 generic.go:356] "Generic (PLEG): container finished" podID="22a07b1d-71c6-46ef-bdd9-ff380101bbc2" containerID="43421128add6b81c554fa443de0797be2fde33e35af32fae2a31666cbd8a48ff" exitCode=6
Jul 17 21:40:52.487744 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:52.487699    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl" event={"ID":"22a07b1d-71c6-46ef-bdd9-ff380101bbc2","Type":"ContainerDied","Data":"43421128add6b81c554fa443de0797be2fde33e35af32fae2a31666cbd8a48ff"}
Jul 17 21:40:52.488194 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:52.487761    2569 scope.go:117] "RemoveContainer" containerID="3636dd2f890ec03d51beb4c8229f7cae051e5e889767c29ed35832e7fb1a2c2b"
Jul 17 21:40:52.488194 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:52.487976    2569 scope.go:117] "RemoveContainer" containerID="43421128add6b81c554fa443de0797be2fde33e35af32fae2a31666cbd8a48ff"
Jul 17 21:40:52.495305 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:40:52.488618    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-v5hjl_odh-ai-gateway-infra(22a07b1d-71c6-46ef-bdd9-ff380101bbc2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl" podUID="22a07b1d-71c6-46ef-bdd9-ff380101bbc2"
Jul 17 21:40:53.495957 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:40:53.495922    2569 scope.go:117] "RemoveContainer" containerID="43421128add6b81c554fa443de0797be2fde33e35af32fae2a31666cbd8a48ff"
Jul 17 21:40:53.496392 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:40:53.496133    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-v5hjl_odh-ai-gateway-infra(22a07b1d-71c6-46ef-bdd9-ff380101bbc2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl" podUID="22a07b1d-71c6-46ef-bdd9-ff380101bbc2"
Jul 17 21:41:03.651048 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:03.650958    2569 scope.go:117] "RemoveContainer" containerID="a3e60df4606ba8b19abff8be83b9e264fcde330c840f784033547a6aff74a67c"
Jul 17 21:41:04.560129 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:04.560042    2569 generic.go:356] "Generic (PLEG): container finished" podID="bb5a4983-99c9-49bd-a96c-343ce782192e" containerID="66e4519c4e3b191c74258e7b70f372c4ed9b25386f2b308c2858c465faef11a3" exitCode=7
Jul 17 21:41:04.560129 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:04.560090    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p" event={"ID":"bb5a4983-99c9-49bd-a96c-343ce782192e","Type":"ContainerDied","Data":"66e4519c4e3b191c74258e7b70f372c4ed9b25386f2b308c2858c465faef11a3"}
Jul 17 21:41:04.560328 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:04.560139    2569 scope.go:117] "RemoveContainer" containerID="a3e60df4606ba8b19abff8be83b9e264fcde330c840f784033547a6aff74a67c"
Jul 17 21:41:04.560469 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:04.560444    2569 scope.go:117] "RemoveContainer" containerID="66e4519c4e3b191c74258e7b70f372c4ed9b25386f2b308c2858c465faef11a3"
Jul 17 21:41:04.560675 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:41:04.560655    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p_odh-ai-gateway-infra(bb5a4983-99c9-49bd-a96c-343ce782192e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p" podUID="bb5a4983-99c9-49bd-a96c-343ce782192e"
Jul 17 21:41:04.657114 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:04.657084    2569 scope.go:117] "RemoveContainer" containerID="43421128add6b81c554fa443de0797be2fde33e35af32fae2a31666cbd8a48ff"
Jul 17 21:41:05.568780 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:05.568691    2569 generic.go:356] "Generic (PLEG): container finished" podID="22a07b1d-71c6-46ef-bdd9-ff380101bbc2" containerID="17a89b2f3d1dc31dc3ce110ddcd0352d9792023b5578b648bb82424321574f5b" exitCode=6
Jul 17 21:41:05.568780 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:05.568759    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl" event={"ID":"22a07b1d-71c6-46ef-bdd9-ff380101bbc2","Type":"ContainerDied","Data":"17a89b2f3d1dc31dc3ce110ddcd0352d9792023b5578b648bb82424321574f5b"}
Jul 17 21:41:05.569011 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:05.568819    2569 scope.go:117] "RemoveContainer" containerID="43421128add6b81c554fa443de0797be2fde33e35af32fae2a31666cbd8a48ff"
Jul 17 21:41:05.569158 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:05.569141    2569 scope.go:117] "RemoveContainer" containerID="17a89b2f3d1dc31dc3ce110ddcd0352d9792023b5578b648bb82424321574f5b"
Jul 17 21:41:05.569376 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:41:05.569357    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-v5hjl_odh-ai-gateway-infra(22a07b1d-71c6-46ef-bdd9-ff380101bbc2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl" podUID="22a07b1d-71c6-46ef-bdd9-ff380101bbc2"
Jul 17 21:41:05.584998 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:05.584955    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p"]
Jul 17 21:41:05.724178 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:05.724148    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p"
Jul 17 21:41:05.770112 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:05.770082    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5a4983-99c9-49bd-a96c-343ce782192e-maas-api-service-ca\") pod \"bb5a4983-99c9-49bd-a96c-343ce782192e\" (UID: \"bb5a4983-99c9-49bd-a96c-343ce782192e\") "
Jul 17 21:41:05.770495 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:05.770476    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5a4983-99c9-49bd-a96c-343ce782192e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bb5a4983-99c9-49bd-a96c-343ce782192e" (UID: "bb5a4983-99c9-49bd-a96c-343ce782192e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:41:05.870739 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:05.870708    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5a4983-99c9-49bd-a96c-343ce782192e-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:41:06.576134 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:06.576097    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p"
Jul 17 21:41:06.576334 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:06.576096    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p" event={"ID":"bb5a4983-99c9-49bd-a96c-343ce782192e","Type":"ContainerDied","Data":"b415486a9a9fde92d98160a0b6cf2f99f1205b80117e9cb2e285d46fc800c15f"}
Jul 17 21:41:06.576334 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:06.576289    2569 scope.go:117] "RemoveContainer" containerID="66e4519c4e3b191c74258e7b70f372c4ed9b25386f2b308c2858c465faef11a3"
Jul 17 21:41:06.597859 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:06.597834    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl"]
Jul 17 21:41:06.603959 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:06.603930    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p"]
Jul 17 21:41:06.609697 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:06.609665    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5jl7p"]
Jul 17 21:41:06.658661 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:06.658528    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb5a4983-99c9-49bd-a96c-343ce782192e" path="/var/lib/kubelet/pods/bb5a4983-99c9-49bd-a96c-343ce782192e/volumes"
Jul 17 21:41:06.741148 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:06.741125    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl"
Jul 17 21:41:06.777331 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:06.777300    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a07b1d-71c6-46ef-bdd9-ff380101bbc2-maas-api-service-ca\") pod \"22a07b1d-71c6-46ef-bdd9-ff380101bbc2\" (UID: \"22a07b1d-71c6-46ef-bdd9-ff380101bbc2\") "
Jul 17 21:41:06.777687 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:06.777660    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a07b1d-71c6-46ef-bdd9-ff380101bbc2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22a07b1d-71c6-46ef-bdd9-ff380101bbc2" (UID: "22a07b1d-71c6-46ef-bdd9-ff380101bbc2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:41:06.878118 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:06.878086    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a07b1d-71c6-46ef-bdd9-ff380101bbc2-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:41:07.585648 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:07.585611    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl" event={"ID":"22a07b1d-71c6-46ef-bdd9-ff380101bbc2","Type":"ContainerDied","Data":"1a523d428e34edfabac416c7c2084fb2173a3611949a1eaa35c542946f028e47"}
Jul 17 21:41:07.585648 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:07.585642    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl"
Jul 17 21:41:07.585882 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:07.585678    2569 scope.go:117] "RemoveContainer" containerID="17a89b2f3d1dc31dc3ce110ddcd0352d9792023b5578b648bb82424321574f5b"
Jul 17 21:41:07.609578 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:07.609548    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl"]
Jul 17 21:41:07.615298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:07.615273    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5hjl"]
Jul 17 21:41:08.657079 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.657045    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22a07b1d-71c6-46ef-bdd9-ff380101bbc2" path="/var/lib/kubelet/pods/22a07b1d-71c6-46ef-bdd9-ff380101bbc2/volumes"
Jul 17 21:41:08.788378 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.788346    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx"]
Jul 17 21:41:08.788763 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.788749    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb5a4983-99c9-49bd-a96c-343ce782192e" containerName="revoke-keys"
Jul 17 21:41:08.788807 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.788764    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb5a4983-99c9-49bd-a96c-343ce782192e" containerName="revoke-keys"
Jul 17 21:41:08.788807 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.788782    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb5a4983-99c9-49bd-a96c-343ce782192e" containerName="revoke-keys"
Jul 17 21:41:08.788807 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.788788    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb5a4983-99c9-49bd-a96c-343ce782192e" containerName="revoke-keys"
Jul 17 21:41:08.788807 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.788794    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a07b1d-71c6-46ef-bdd9-ff380101bbc2" containerName="revoke-keys"
Jul 17 21:41:08.788807 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.788799    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a07b1d-71c6-46ef-bdd9-ff380101bbc2" containerName="revoke-keys"
Jul 17 21:41:08.788807 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.788805    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a07b1d-71c6-46ef-bdd9-ff380101bbc2" containerName="revoke-keys"
Jul 17 21:41:08.788982 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.788812    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a07b1d-71c6-46ef-bdd9-ff380101bbc2" containerName="revoke-keys"
Jul 17 21:41:08.788982 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.788822    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a07b1d-71c6-46ef-bdd9-ff380101bbc2" containerName="revoke-keys"
Jul 17 21:41:08.788982 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.788828    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a07b1d-71c6-46ef-bdd9-ff380101bbc2" containerName="revoke-keys"
Jul 17 21:41:08.788982 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.788883    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb5a4983-99c9-49bd-a96c-343ce782192e" containerName="revoke-keys"
Jul 17 21:41:08.788982 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.788894    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a07b1d-71c6-46ef-bdd9-ff380101bbc2" containerName="revoke-keys"
Jul 17 21:41:08.788982 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.788900    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a07b1d-71c6-46ef-bdd9-ff380101bbc2" containerName="revoke-keys"
Jul 17 21:41:08.788982 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.788907    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb5a4983-99c9-49bd-a96c-343ce782192e" containerName="revoke-keys"
Jul 17 21:41:08.793304 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.793287    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx"
Jul 17 21:41:08.796299 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.796280    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:41:08.796580 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.796565    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:41:08.803974 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.803948    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx"]
Jul 17 21:41:08.895277 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.895240    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb82df5b-042d-43f8-8971-458700ea207e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx\" (UID: \"bb82df5b-042d-43f8-8971-458700ea207e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx"
Jul 17 21:41:08.996298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.996216    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb82df5b-042d-43f8-8971-458700ea207e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx\" (UID: \"bb82df5b-042d-43f8-8971-458700ea207e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx"
Jul 17 21:41:08.996869 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:08.996850    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb82df5b-042d-43f8-8971-458700ea207e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx\" (UID: \"bb82df5b-042d-43f8-8971-458700ea207e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx"
Jul 17 21:41:09.104741 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:09.104699    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx"
Jul 17 21:41:09.241545 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:09.241509    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx"]
Jul 17 21:41:09.243025 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:41:09.242994    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb82df5b_042d_43f8_8971_458700ea207e.slice/crio-418e7f385ab503eaf33a3b4cdcd3020533ed9d73141e1e3ce9723857b8cf3dff WatchSource:0}: Error finding container 418e7f385ab503eaf33a3b4cdcd3020533ed9d73141e1e3ce9723857b8cf3dff: Status 404 returned error can't find the container with id 418e7f385ab503eaf33a3b4cdcd3020533ed9d73141e1e3ce9723857b8cf3dff
Jul 17 21:41:09.600995 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:09.600951    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx" event={"ID":"bb82df5b-042d-43f8-8971-458700ea207e","Type":"ContainerStarted","Data":"60ea44d06c55b84b2155f74f6277301ea6fec567ee5390b00a77d6c7ee021036"}
Jul 17 21:41:09.601170 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:09.601038    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx" event={"ID":"bb82df5b-042d-43f8-8971-458700ea207e","Type":"ContainerStarted","Data":"418e7f385ab503eaf33a3b4cdcd3020533ed9d73141e1e3ce9723857b8cf3dff"}
Jul 17 21:41:09.616737 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:09.616688    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx" podStartSLOduration=1.6166742539999999 podStartE2EDuration="1.616674254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:41:09.615250367 +0000 UTC m=+1983.374817072" watchObservedRunningTime="2026-07-17 21:41:09.616674254 +0000 UTC m=+1983.376240948"
Jul 17 21:41:10.609075 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:10.609038    2569 generic.go:356] "Generic (PLEG): container finished" podID="bb82df5b-042d-43f8-8971-458700ea207e" containerID="60ea44d06c55b84b2155f74f6277301ea6fec567ee5390b00a77d6c7ee021036" exitCode=6
Jul 17 21:41:10.609452 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:10.609077    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx" event={"ID":"bb82df5b-042d-43f8-8971-458700ea207e","Type":"ContainerDied","Data":"60ea44d06c55b84b2155f74f6277301ea6fec567ee5390b00a77d6c7ee021036"}
Jul 17 21:41:10.609452 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:10.609322    2569 scope.go:117] "RemoveContainer" containerID="60ea44d06c55b84b2155f74f6277301ea6fec567ee5390b00a77d6c7ee021036"
Jul 17 21:41:11.618625 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:11.618585    2569 generic.go:356] "Generic (PLEG): container finished" podID="bb82df5b-042d-43f8-8971-458700ea207e" containerID="00c84757037bde8d556cfc521530face7f8753f45c21a497f4125804a9cd8c39" exitCode=6
Jul 17 21:41:11.619079 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:11.618672    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx" event={"ID":"bb82df5b-042d-43f8-8971-458700ea207e","Type":"ContainerDied","Data":"00c84757037bde8d556cfc521530face7f8753f45c21a497f4125804a9cd8c39"}
Jul 17 21:41:11.619079 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:11.618738    2569 scope.go:117] "RemoveContainer" containerID="60ea44d06c55b84b2155f74f6277301ea6fec567ee5390b00a77d6c7ee021036"
Jul 17 21:41:11.619079 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:11.618971    2569 scope.go:117] "RemoveContainer" containerID="00c84757037bde8d556cfc521530face7f8753f45c21a497f4125804a9cd8c39"
Jul 17 21:41:11.619225 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:41:11.619201    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx_odh-ai-gateway-infra(bb82df5b-042d-43f8-8971-458700ea207e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx" podUID="bb82df5b-042d-43f8-8971-458700ea207e"
Jul 17 21:41:12.628121 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:12.628097    2569 scope.go:117] "RemoveContainer" containerID="00c84757037bde8d556cfc521530face7f8753f45c21a497f4125804a9cd8c39"
Jul 17 21:41:12.628510 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:41:12.628275    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx_odh-ai-gateway-infra(bb82df5b-042d-43f8-8971-458700ea207e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx" podUID="bb82df5b-042d-43f8-8971-458700ea207e"
Jul 17 21:41:26.653203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:26.653173    2569 scope.go:117] "RemoveContainer" containerID="00c84757037bde8d556cfc521530face7f8753f45c21a497f4125804a9cd8c39"
Jul 17 21:41:27.713328 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:27.713295    2569 generic.go:356] "Generic (PLEG): container finished" podID="bb82df5b-042d-43f8-8971-458700ea207e" containerID="772bbad6bc62abd56e32f2d95c7d7dc0c56b8c6b96d6288befd7bd48a29aa0a9" exitCode=6
Jul 17 21:41:27.713780 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:27.713368    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx" event={"ID":"bb82df5b-042d-43f8-8971-458700ea207e","Type":"ContainerDied","Data":"772bbad6bc62abd56e32f2d95c7d7dc0c56b8c6b96d6288befd7bd48a29aa0a9"}
Jul 17 21:41:27.713780 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:27.713454    2569 scope.go:117] "RemoveContainer" containerID="00c84757037bde8d556cfc521530face7f8753f45c21a497f4125804a9cd8c39"
Jul 17 21:41:27.713780 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:27.713723    2569 scope.go:117] "RemoveContainer" containerID="772bbad6bc62abd56e32f2d95c7d7dc0c56b8c6b96d6288befd7bd48a29aa0a9"
Jul 17 21:41:27.714003 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:41:27.713944    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx_odh-ai-gateway-infra(bb82df5b-042d-43f8-8971-458700ea207e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx" podUID="bb82df5b-042d-43f8-8971-458700ea207e"
Jul 17 21:41:28.739537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:28.739503    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx"]
Jul 17 21:41:28.883334 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:28.883299    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx"
Jul 17 21:41:28.976427 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:28.976370    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb82df5b-042d-43f8-8971-458700ea207e-maas-api-service-ca\") pod \"bb82df5b-042d-43f8-8971-458700ea207e\" (UID: \"bb82df5b-042d-43f8-8971-458700ea207e\") "
Jul 17 21:41:28.976756 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:28.976731    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb82df5b-042d-43f8-8971-458700ea207e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bb82df5b-042d-43f8-8971-458700ea207e" (UID: "bb82df5b-042d-43f8-8971-458700ea207e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:41:29.077880 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:29.077784    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb82df5b-042d-43f8-8971-458700ea207e-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:41:29.727553 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:29.727521    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx"
Jul 17 21:41:29.727740 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:29.727551    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx" event={"ID":"bb82df5b-042d-43f8-8971-458700ea207e","Type":"ContainerDied","Data":"418e7f385ab503eaf33a3b4cdcd3020533ed9d73141e1e3ce9723857b8cf3dff"}
Jul 17 21:41:29.727740 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:29.727615    2569 scope.go:117] "RemoveContainer" containerID="772bbad6bc62abd56e32f2d95c7d7dc0c56b8c6b96d6288befd7bd48a29aa0a9"
Jul 17 21:41:29.750605 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:29.750576    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx"]
Jul 17 21:41:29.754190 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:29.754166    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-58jcx"]
Jul 17 21:41:30.657795 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:30.657764    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb82df5b-042d-43f8-8971-458700ea207e" path="/var/lib/kubelet/pods/bb82df5b-042d-43f8-8971-458700ea207e/volumes"
Jul 17 21:41:36.873839 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:36.873798    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f"]
Jul 17 21:41:36.874278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:36.874185    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb82df5b-042d-43f8-8971-458700ea207e" containerName="revoke-keys"
Jul 17 21:41:36.874278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:36.874196    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb82df5b-042d-43f8-8971-458700ea207e" containerName="revoke-keys"
Jul 17 21:41:36.874278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:36.874218    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb5a4983-99c9-49bd-a96c-343ce782192e" containerName="revoke-keys"
Jul 17 21:41:36.874278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:36.874223    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb5a4983-99c9-49bd-a96c-343ce782192e" containerName="revoke-keys"
Jul 17 21:41:36.874278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:36.874230    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb82df5b-042d-43f8-8971-458700ea207e" containerName="revoke-keys"
Jul 17 21:41:36.874278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:36.874235    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb82df5b-042d-43f8-8971-458700ea207e" containerName="revoke-keys"
Jul 17 21:41:36.874543 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:36.874311    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb82df5b-042d-43f8-8971-458700ea207e" containerName="revoke-keys"
Jul 17 21:41:36.874543 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:36.874317    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a07b1d-71c6-46ef-bdd9-ff380101bbc2" containerName="revoke-keys"
Jul 17 21:41:36.874543 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:36.874324    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb82df5b-042d-43f8-8971-458700ea207e" containerName="revoke-keys"
Jul 17 21:41:36.874543 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:36.874330    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb5a4983-99c9-49bd-a96c-343ce782192e" containerName="revoke-keys"
Jul 17 21:41:36.877667 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:36.877649    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f"
Jul 17 21:41:36.879627 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:36.879598    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:41:36.879732 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:36.879685    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:41:36.883828 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:36.883803    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f"]
Jul 17 21:41:37.045071 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:37.045037    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f50c2517-f0f0-4f08-9e5e-ff033b77548e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f\" (UID: \"f50c2517-f0f0-4f08-9e5e-ff033b77548e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f"
Jul 17 21:41:37.146499 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:37.146382    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f50c2517-f0f0-4f08-9e5e-ff033b77548e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f\" (UID: \"f50c2517-f0f0-4f08-9e5e-ff033b77548e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f"
Jul 17 21:41:37.147035 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:37.147014    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f50c2517-f0f0-4f08-9e5e-ff033b77548e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f\" (UID: \"f50c2517-f0f0-4f08-9e5e-ff033b77548e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f"
Jul 17 21:41:37.189428 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:37.189373    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f"
Jul 17 21:41:37.345697 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:37.345672    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f"]
Jul 17 21:41:37.347208 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:41:37.347181    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf50c2517_f0f0_4f08_9e5e_ff033b77548e.slice/crio-e9cb79b48757faadb3490c2139c7a6866dfe081c633119b8e75222cd8ee029fe WatchSource:0}: Error finding container e9cb79b48757faadb3490c2139c7a6866dfe081c633119b8e75222cd8ee029fe: Status 404 returned error can't find the container with id e9cb79b48757faadb3490c2139c7a6866dfe081c633119b8e75222cd8ee029fe
Jul 17 21:41:37.778683 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:37.778592    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f" event={"ID":"f50c2517-f0f0-4f08-9e5e-ff033b77548e","Type":"ContainerStarted","Data":"e7d69fffbc80bf68358886fce3a2737810043f82bb7d97793e0b4b958e4f0b8d"}
Jul 17 21:41:37.778683 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:37.778656    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f" event={"ID":"f50c2517-f0f0-4f08-9e5e-ff033b77548e","Type":"ContainerStarted","Data":"e9cb79b48757faadb3490c2139c7a6866dfe081c633119b8e75222cd8ee029fe"}
Jul 17 21:41:38.785622 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:38.785588    2569 generic.go:356] "Generic (PLEG): container finished" podID="f50c2517-f0f0-4f08-9e5e-ff033b77548e" containerID="e7d69fffbc80bf68358886fce3a2737810043f82bb7d97793e0b4b958e4f0b8d" exitCode=7
Jul 17 21:41:38.786023 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:38.785638    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f" event={"ID":"f50c2517-f0f0-4f08-9e5e-ff033b77548e","Type":"ContainerDied","Data":"e7d69fffbc80bf68358886fce3a2737810043f82bb7d97793e0b4b958e4f0b8d"}
Jul 17 21:41:38.786023 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:38.785856    2569 scope.go:117] "RemoveContainer" containerID="e7d69fffbc80bf68358886fce3a2737810043f82bb7d97793e0b4b958e4f0b8d"
Jul 17 21:41:39.793583 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:39.793489    2569 generic.go:356] "Generic (PLEG): container finished" podID="f50c2517-f0f0-4f08-9e5e-ff033b77548e" containerID="e7ef3101098f3d9f23fa1e9d03f0e10ab68098ae4cfbf399ff4c60f17f7907a5" exitCode=7
Jul 17 21:41:39.793583 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:39.793556    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f" event={"ID":"f50c2517-f0f0-4f08-9e5e-ff033b77548e","Type":"ContainerDied","Data":"e7ef3101098f3d9f23fa1e9d03f0e10ab68098ae4cfbf399ff4c60f17f7907a5"}
Jul 17 21:41:39.794039 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:39.793621    2569 scope.go:117] "RemoveContainer" containerID="e7d69fffbc80bf68358886fce3a2737810043f82bb7d97793e0b4b958e4f0b8d"
Jul 17 21:41:39.794039 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:39.793799    2569 scope.go:117] "RemoveContainer" containerID="e7ef3101098f3d9f23fa1e9d03f0e10ab68098ae4cfbf399ff4c60f17f7907a5"
Jul 17 21:41:39.794111 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:41:39.794056    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f_odh-ai-gateway-infra(f50c2517-f0f0-4f08-9e5e-ff033b77548e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f" podUID="f50c2517-f0f0-4f08-9e5e-ff033b77548e"
Jul 17 21:41:39.985126 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:39.985090    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp"]
Jul 17 21:41:39.985526 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:39.985511    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb82df5b-042d-43f8-8971-458700ea207e" containerName="revoke-keys"
Jul 17 21:41:39.985526 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:39.985525    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb82df5b-042d-43f8-8971-458700ea207e" containerName="revoke-keys"
Jul 17 21:41:39.985687 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:39.985611    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb82df5b-042d-43f8-8971-458700ea207e" containerName="revoke-keys"
Jul 17 21:41:39.989884 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:39.989864    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp"
Jul 17 21:41:39.996204 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:39.996176    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp"]
Jul 17 21:41:40.174193 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:40.174151    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d740a827-011f-434e-beaa-e6f67eba77db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-k7ltp\" (UID: \"d740a827-011f-434e-beaa-e6f67eba77db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp"
Jul 17 21:41:40.275062 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:40.275020    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d740a827-011f-434e-beaa-e6f67eba77db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-k7ltp\" (UID: \"d740a827-011f-434e-beaa-e6f67eba77db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp"
Jul 17 21:41:40.275704 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:40.275683    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d740a827-011f-434e-beaa-e6f67eba77db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-k7ltp\" (UID: \"d740a827-011f-434e-beaa-e6f67eba77db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp"
Jul 17 21:41:40.302040 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:40.302007    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp"
Jul 17 21:41:40.648592 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:40.648567    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp"]
Jul 17 21:41:40.650538 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:41:40.650505    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd740a827_011f_434e_beaa_e6f67eba77db.slice/crio-126eaea6a2c15bbd60c991fc046ba4d36b419a2d4a873fd8f5e0f696cb1c0e0e WatchSource:0}: Error finding container 126eaea6a2c15bbd60c991fc046ba4d36b419a2d4a873fd8f5e0f696cb1c0e0e: Status 404 returned error can't find the container with id 126eaea6a2c15bbd60c991fc046ba4d36b419a2d4a873fd8f5e0f696cb1c0e0e
Jul 17 21:41:40.800993 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:40.800958    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp" event={"ID":"d740a827-011f-434e-beaa-e6f67eba77db","Type":"ContainerStarted","Data":"126eaea6a2c15bbd60c991fc046ba4d36b419a2d4a873fd8f5e0f696cb1c0e0e"}
Jul 17 21:41:40.802902 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:40.802880    2569 scope.go:117] "RemoveContainer" containerID="e7ef3101098f3d9f23fa1e9d03f0e10ab68098ae4cfbf399ff4c60f17f7907a5"
Jul 17 21:41:40.803099 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:41:40.803081    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f_odh-ai-gateway-infra(f50c2517-f0f0-4f08-9e5e-ff033b77548e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f" podUID="f50c2517-f0f0-4f08-9e5e-ff033b77548e"
Jul 17 21:41:41.809836 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:41.809749    2569 generic.go:356] "Generic (PLEG): container finished" podID="d740a827-011f-434e-beaa-e6f67eba77db" containerID="480faf5d2ec769e91f474e021e457533c49943336e13c3aafda97ffaaa9edf02" exitCode=6
Jul 17 21:41:41.810199 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:41.809837    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp" event={"ID":"d740a827-011f-434e-beaa-e6f67eba77db","Type":"ContainerDied","Data":"480faf5d2ec769e91f474e021e457533c49943336e13c3aafda97ffaaa9edf02"}
Jul 17 21:41:41.810199 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:41.810011    2569 scope.go:117] "RemoveContainer" containerID="480faf5d2ec769e91f474e021e457533c49943336e13c3aafda97ffaaa9edf02"
Jul 17 21:41:42.816836 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:42.816744    2569 generic.go:356] "Generic (PLEG): container finished" podID="d740a827-011f-434e-beaa-e6f67eba77db" containerID="71d39d45f93527456b22f363c40561b639c683f0e462609e8c5c41a5a2ccaa27" exitCode=6
Jul 17 21:41:42.817267 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:42.816829    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp" event={"ID":"d740a827-011f-434e-beaa-e6f67eba77db","Type":"ContainerDied","Data":"71d39d45f93527456b22f363c40561b639c683f0e462609e8c5c41a5a2ccaa27"}
Jul 17 21:41:42.817267 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:42.816883    2569 scope.go:117] "RemoveContainer" containerID="480faf5d2ec769e91f474e021e457533c49943336e13c3aafda97ffaaa9edf02"
Jul 17 21:41:42.817267 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:42.817087    2569 scope.go:117] "RemoveContainer" containerID="71d39d45f93527456b22f363c40561b639c683f0e462609e8c5c41a5a2ccaa27"
Jul 17 21:41:42.817435 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:41:42.817278    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-k7ltp_odh-ai-gateway-infra(d740a827-011f-434e-beaa-e6f67eba77db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp" podUID="d740a827-011f-434e-beaa-e6f67eba77db"
Jul 17 21:41:43.824168 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:43.824125    2569 scope.go:117] "RemoveContainer" containerID="71d39d45f93527456b22f363c40561b639c683f0e462609e8c5c41a5a2ccaa27"
Jul 17 21:41:43.824574 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:41:43.824340    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-k7ltp_odh-ai-gateway-infra(d740a827-011f-434e-beaa-e6f67eba77db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp" podUID="d740a827-011f-434e-beaa-e6f67eba77db"
Jul 17 21:41:55.650889 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:55.650851    2569 scope.go:117] "RemoveContainer" containerID="e7ef3101098f3d9f23fa1e9d03f0e10ab68098ae4cfbf399ff4c60f17f7907a5"
Jul 17 21:41:56.902576 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:56.902541    2569 generic.go:356] "Generic (PLEG): container finished" podID="f50c2517-f0f0-4f08-9e5e-ff033b77548e" containerID="0a9934c181c5dacf2915182d24e320b42be74f94ce10395d77a0415ad7f447b5" exitCode=7
Jul 17 21:41:56.902963 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:56.902599    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f" event={"ID":"f50c2517-f0f0-4f08-9e5e-ff033b77548e","Type":"ContainerDied","Data":"0a9934c181c5dacf2915182d24e320b42be74f94ce10395d77a0415ad7f447b5"}
Jul 17 21:41:56.902963 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:56.902650    2569 scope.go:117] "RemoveContainer" containerID="e7ef3101098f3d9f23fa1e9d03f0e10ab68098ae4cfbf399ff4c60f17f7907a5"
Jul 17 21:41:56.902963 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:56.902915    2569 scope.go:117] "RemoveContainer" containerID="0a9934c181c5dacf2915182d24e320b42be74f94ce10395d77a0415ad7f447b5"
Jul 17 21:41:56.903171 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:41:56.903124    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f_odh-ai-gateway-infra(f50c2517-f0f0-4f08-9e5e-ff033b77548e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f" podUID="f50c2517-f0f0-4f08-9e5e-ff033b77548e"
Jul 17 21:41:57.927849 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:57.927815    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f"]
Jul 17 21:41:58.066368 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:58.066343    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f"
Jul 17 21:41:58.122127 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:58.122090    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f50c2517-f0f0-4f08-9e5e-ff033b77548e-maas-api-service-ca\") pod \"f50c2517-f0f0-4f08-9e5e-ff033b77548e\" (UID: \"f50c2517-f0f0-4f08-9e5e-ff033b77548e\") "
Jul 17 21:41:58.122495 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:58.122473    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f50c2517-f0f0-4f08-9e5e-ff033b77548e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f50c2517-f0f0-4f08-9e5e-ff033b77548e" (UID: "f50c2517-f0f0-4f08-9e5e-ff033b77548e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:41:58.223597 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:58.223508    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f50c2517-f0f0-4f08-9e5e-ff033b77548e-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:41:58.651217 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:58.651188    2569 scope.go:117] "RemoveContainer" containerID="71d39d45f93527456b22f363c40561b639c683f0e462609e8c5c41a5a2ccaa27"
Jul 17 21:41:58.917628 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:58.917540    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f"
Jul 17 21:41:58.917628 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:58.917555    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f" event={"ID":"f50c2517-f0f0-4f08-9e5e-ff033b77548e","Type":"ContainerDied","Data":"e9cb79b48757faadb3490c2139c7a6866dfe081c633119b8e75222cd8ee029fe"}
Jul 17 21:41:58.917628 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:58.917616    2569 scope.go:117] "RemoveContainer" containerID="0a9934c181c5dacf2915182d24e320b42be74f94ce10395d77a0415ad7f447b5"
Jul 17 21:41:58.933990 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:58.933961    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f"]
Jul 17 21:41:58.939229 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:58.939207    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-fh99f"]
Jul 17 21:41:59.925276 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:59.925247    2569 generic.go:356] "Generic (PLEG): container finished" podID="d740a827-011f-434e-beaa-e6f67eba77db" containerID="353a454b704c957ff7f6b175ecd9efd95ed95ba9b21ec1faebb69c3fe24acb03" exitCode=6
Jul 17 21:41:59.925498 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:59.925321    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp" event={"ID":"d740a827-011f-434e-beaa-e6f67eba77db","Type":"ContainerDied","Data":"353a454b704c957ff7f6b175ecd9efd95ed95ba9b21ec1faebb69c3fe24acb03"}
Jul 17 21:41:59.925498 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:59.925384    2569 scope.go:117] "RemoveContainer" containerID="71d39d45f93527456b22f363c40561b639c683f0e462609e8c5c41a5a2ccaa27"
Jul 17 21:41:59.925688 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:41:59.925667    2569 scope.go:117] "RemoveContainer" containerID="353a454b704c957ff7f6b175ecd9efd95ed95ba9b21ec1faebb69c3fe24acb03"
Jul 17 21:41:59.925955 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:41:59.925930    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-k7ltp_odh-ai-gateway-infra(d740a827-011f-434e-beaa-e6f67eba77db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp" podUID="d740a827-011f-434e-beaa-e6f67eba77db"
Jul 17 21:42:00.657006 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:00.656976    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f50c2517-f0f0-4f08-9e5e-ff033b77548e" path="/var/lib/kubelet/pods/f50c2517-f0f0-4f08-9e5e-ff033b77548e/volumes"
Jul 17 21:42:00.953560 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:00.953476    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp"]
Jul 17 21:42:01.097245 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:01.097217    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp"
Jul 17 21:42:01.148388 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:01.148358    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d740a827-011f-434e-beaa-e6f67eba77db-maas-api-service-ca\") pod \"d740a827-011f-434e-beaa-e6f67eba77db\" (UID: \"d740a827-011f-434e-beaa-e6f67eba77db\") "
Jul 17 21:42:01.148758 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:01.148734    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d740a827-011f-434e-beaa-e6f67eba77db-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d740a827-011f-434e-beaa-e6f67eba77db" (UID: "d740a827-011f-434e-beaa-e6f67eba77db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:42:01.249446 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:01.249332    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d740a827-011f-434e-beaa-e6f67eba77db-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:42:01.947526 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:01.947496    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp"
Jul 17 21:42:01.947526 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:01.947512    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp" event={"ID":"d740a827-011f-434e-beaa-e6f67eba77db","Type":"ContainerDied","Data":"126eaea6a2c15bbd60c991fc046ba4d36b419a2d4a873fd8f5e0f696cb1c0e0e"}
Jul 17 21:42:01.948167 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:01.947567    2569 scope.go:117] "RemoveContainer" containerID="353a454b704c957ff7f6b175ecd9efd95ed95ba9b21ec1faebb69c3fe24acb03"
Jul 17 21:42:01.988068 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:01.988033    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp"]
Jul 17 21:42:02.002181 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:02.002150    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-k7ltp"]
Jul 17 21:42:02.656982 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:02.656951    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d740a827-011f-434e-beaa-e6f67eba77db" path="/var/lib/kubelet/pods/d740a827-011f-434e-beaa-e6f67eba77db/volumes"
Jul 17 21:42:08.828790 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.828750    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm"]
Jul 17 21:42:08.829226 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.829168    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f50c2517-f0f0-4f08-9e5e-ff033b77548e" containerName="revoke-keys"
Jul 17 21:42:08.829226 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.829180    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f50c2517-f0f0-4f08-9e5e-ff033b77548e" containerName="revoke-keys"
Jul 17 21:42:08.829226 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.829196    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f50c2517-f0f0-4f08-9e5e-ff033b77548e" containerName="revoke-keys"
Jul 17 21:42:08.829226 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.829202    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f50c2517-f0f0-4f08-9e5e-ff033b77548e" containerName="revoke-keys"
Jul 17 21:42:08.829226 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.829209    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d740a827-011f-434e-beaa-e6f67eba77db" containerName="revoke-keys"
Jul 17 21:42:08.829226 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.829215    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d740a827-011f-434e-beaa-e6f67eba77db" containerName="revoke-keys"
Jul 17 21:42:08.829226 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.829220    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d740a827-011f-434e-beaa-e6f67eba77db" containerName="revoke-keys"
Jul 17 21:42:08.829226 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.829226    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d740a827-011f-434e-beaa-e6f67eba77db" containerName="revoke-keys"
Jul 17 21:42:08.829226 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.829232    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f50c2517-f0f0-4f08-9e5e-ff033b77548e" containerName="revoke-keys"
Jul 17 21:42:08.829566 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.829237    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f50c2517-f0f0-4f08-9e5e-ff033b77548e" containerName="revoke-keys"
Jul 17 21:42:08.829566 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.829250    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d740a827-011f-434e-beaa-e6f67eba77db" containerName="revoke-keys"
Jul 17 21:42:08.829566 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.829254    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d740a827-011f-434e-beaa-e6f67eba77db" containerName="revoke-keys"
Jul 17 21:42:08.829566 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.829314    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f50c2517-f0f0-4f08-9e5e-ff033b77548e" containerName="revoke-keys"
Jul 17 21:42:08.829566 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.829321    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f50c2517-f0f0-4f08-9e5e-ff033b77548e" containerName="revoke-keys"
Jul 17 21:42:08.829566 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.829328    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d740a827-011f-434e-beaa-e6f67eba77db" containerName="revoke-keys"
Jul 17 21:42:08.829566 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.829334    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f50c2517-f0f0-4f08-9e5e-ff033b77548e" containerName="revoke-keys"
Jul 17 21:42:08.829566 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.829339    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d740a827-011f-434e-beaa-e6f67eba77db" containerName="revoke-keys"
Jul 17 21:42:08.829566 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.829343    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d740a827-011f-434e-beaa-e6f67eba77db" containerName="revoke-keys"
Jul 17 21:42:08.833839 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.833821    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm"
Jul 17 21:42:08.836000 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.835978    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:42:08.836116 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.835978    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:42:08.840472 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.840447    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm"]
Jul 17 21:42:08.916134 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:08.916095    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c36d1fee-cb56-4457-bb7a-7ded8dd08a9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm\" (UID: \"c36d1fee-cb56-4457-bb7a-7ded8dd08a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm"
Jul 17 21:42:09.017060 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:09.017022    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c36d1fee-cb56-4457-bb7a-7ded8dd08a9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm\" (UID: \"c36d1fee-cb56-4457-bb7a-7ded8dd08a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm"
Jul 17 21:42:09.017704 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:09.017683    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c36d1fee-cb56-4457-bb7a-7ded8dd08a9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm\" (UID: \"c36d1fee-cb56-4457-bb7a-7ded8dd08a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm"
Jul 17 21:42:09.145805 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:09.145754    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm"
Jul 17 21:42:09.281708 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:09.281680    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm"]
Jul 17 21:42:09.284177 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:42:09.284144    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36d1fee_cb56_4457_bb7a_7ded8dd08a9b.slice/crio-6be67dd23671f1832019ef708ad065041e1e4a5afc83a149b743b3bfc3ca115b WatchSource:0}: Error finding container 6be67dd23671f1832019ef708ad065041e1e4a5afc83a149b743b3bfc3ca115b: Status 404 returned error can't find the container with id 6be67dd23671f1832019ef708ad065041e1e4a5afc83a149b743b3bfc3ca115b
Jul 17 21:42:09.995631 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:09.995596    2569 generic.go:356] "Generic (PLEG): container finished" podID="c36d1fee-cb56-4457-bb7a-7ded8dd08a9b" containerID="98bf6f89cb7cb039d0c02db815c96995e7e11a2a762da72aff5ff4b9a2591fc2" exitCode=6
Jul 17 21:42:09.996032 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:09.995692    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm" event={"ID":"c36d1fee-cb56-4457-bb7a-7ded8dd08a9b","Type":"ContainerDied","Data":"98bf6f89cb7cb039d0c02db815c96995e7e11a2a762da72aff5ff4b9a2591fc2"}
Jul 17 21:42:09.996032 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:09.995745    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm" event={"ID":"c36d1fee-cb56-4457-bb7a-7ded8dd08a9b","Type":"ContainerStarted","Data":"6be67dd23671f1832019ef708ad065041e1e4a5afc83a149b743b3bfc3ca115b"}
Jul 17 21:42:09.996032 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:09.995896    2569 scope.go:117] "RemoveContainer" containerID="98bf6f89cb7cb039d0c02db815c96995e7e11a2a762da72aff5ff4b9a2591fc2"
Jul 17 21:42:11.002943 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:11.002903    2569 generic.go:356] "Generic (PLEG): container finished" podID="c36d1fee-cb56-4457-bb7a-7ded8dd08a9b" containerID="02eb97ee2004be33352233d963d84290d5d4563fac2c0b3479ebddba11a81da4" exitCode=6
Jul 17 21:42:11.003374 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:11.002988    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm" event={"ID":"c36d1fee-cb56-4457-bb7a-7ded8dd08a9b","Type":"ContainerDied","Data":"02eb97ee2004be33352233d963d84290d5d4563fac2c0b3479ebddba11a81da4"}
Jul 17 21:42:11.003374 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:11.003046    2569 scope.go:117] "RemoveContainer" containerID="98bf6f89cb7cb039d0c02db815c96995e7e11a2a762da72aff5ff4b9a2591fc2"
Jul 17 21:42:11.003374 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:11.003187    2569 scope.go:117] "RemoveContainer" containerID="02eb97ee2004be33352233d963d84290d5d4563fac2c0b3479ebddba11a81da4"
Jul 17 21:42:11.003524 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:42:11.003419    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm_odh-ai-gateway-infra(c36d1fee-cb56-4457-bb7a-7ded8dd08a9b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm" podUID="c36d1fee-cb56-4457-bb7a-7ded8dd08a9b"
Jul 17 21:42:12.010563 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:12.010531    2569 scope.go:117] "RemoveContainer" containerID="02eb97ee2004be33352233d963d84290d5d4563fac2c0b3479ebddba11a81da4"
Jul 17 21:42:12.011036 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:42:12.010790    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm_odh-ai-gateway-infra(c36d1fee-cb56-4457-bb7a-7ded8dd08a9b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm" podUID="c36d1fee-cb56-4457-bb7a-7ded8dd08a9b"
Jul 17 21:42:23.650597 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:23.650563    2569 scope.go:117] "RemoveContainer" containerID="02eb97ee2004be33352233d963d84290d5d4563fac2c0b3479ebddba11a81da4"
Jul 17 21:42:24.080637 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:24.080603    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm" event={"ID":"c36d1fee-cb56-4457-bb7a-7ded8dd08a9b","Type":"ContainerStarted","Data":"c5f2a71bf8ca3fbb16cbe73434505e4d4cb6ddc7ce1687c3154815cb31127f48"}
Jul 17 21:42:24.096781 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:24.096732    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm" podStartSLOduration=16.096717275 podStartE2EDuration="16.096717275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:42:24.093901366 +0000 UTC m=+2057.853468059" watchObservedRunningTime="2026-07-17 21:42:24.096717275 +0000 UTC m=+2057.856283968"
Jul 17 21:42:25.094988 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:25.094953    2569 generic.go:356] "Generic (PLEG): container finished" podID="c36d1fee-cb56-4457-bb7a-7ded8dd08a9b" containerID="c5f2a71bf8ca3fbb16cbe73434505e4d4cb6ddc7ce1687c3154815cb31127f48" exitCode=6
Jul 17 21:42:25.095527 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:25.095007    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm" event={"ID":"c36d1fee-cb56-4457-bb7a-7ded8dd08a9b","Type":"ContainerDied","Data":"c5f2a71bf8ca3fbb16cbe73434505e4d4cb6ddc7ce1687c3154815cb31127f48"}
Jul 17 21:42:25.095527 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:25.095074    2569 scope.go:117] "RemoveContainer" containerID="02eb97ee2004be33352233d963d84290d5d4563fac2c0b3479ebddba11a81da4"
Jul 17 21:42:25.095527 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:25.095334    2569 scope.go:117] "RemoveContainer" containerID="c5f2a71bf8ca3fbb16cbe73434505e4d4cb6ddc7ce1687c3154815cb31127f48"
Jul 17 21:42:25.095712 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:42:25.095571    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm_odh-ai-gateway-infra(c36d1fee-cb56-4457-bb7a-7ded8dd08a9b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm" podUID="c36d1fee-cb56-4457-bb7a-7ded8dd08a9b"
Jul 17 21:42:25.106197 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:25.106173    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm"]
Jul 17 21:42:26.243197 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:26.243167    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm"
Jul 17 21:42:26.376340 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:26.376242    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c36d1fee-cb56-4457-bb7a-7ded8dd08a9b-maas-api-service-ca\") pod \"c36d1fee-cb56-4457-bb7a-7ded8dd08a9b\" (UID: \"c36d1fee-cb56-4457-bb7a-7ded8dd08a9b\") "
Jul 17 21:42:26.376718 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:26.376688    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c36d1fee-cb56-4457-bb7a-7ded8dd08a9b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c36d1fee-cb56-4457-bb7a-7ded8dd08a9b" (UID: "c36d1fee-cb56-4457-bb7a-7ded8dd08a9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:42:26.477441 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:26.477358    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c36d1fee-cb56-4457-bb7a-7ded8dd08a9b-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:42:27.108737 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:27.108702    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm"
Jul 17 21:42:27.108737 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:27.108728    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm" event={"ID":"c36d1fee-cb56-4457-bb7a-7ded8dd08a9b","Type":"ContainerDied","Data":"6be67dd23671f1832019ef708ad065041e1e4a5afc83a149b743b3bfc3ca115b"}
Jul 17 21:42:27.109010 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:27.108783    2569 scope.go:117] "RemoveContainer" containerID="c5f2a71bf8ca3fbb16cbe73434505e4d4cb6ddc7ce1687c3154815cb31127f48"
Jul 17 21:42:27.124279 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:27.124242    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm"]
Jul 17 21:42:27.126636 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:27.126613    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-zgfmm"]
Jul 17 21:42:28.657331 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:28.657297    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c36d1fee-cb56-4457-bb7a-7ded8dd08a9b" path="/var/lib/kubelet/pods/c36d1fee-cb56-4457-bb7a-7ded8dd08a9b/volumes"
Jul 17 21:42:36.925297 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:36.925216    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd"]
Jul 17 21:42:36.925731 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:36.925665    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c36d1fee-cb56-4457-bb7a-7ded8dd08a9b" containerName="revoke-keys"
Jul 17 21:42:36.925731 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:36.925679    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c36d1fee-cb56-4457-bb7a-7ded8dd08a9b" containerName="revoke-keys"
Jul 17 21:42:36.925731 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:36.925706    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c36d1fee-cb56-4457-bb7a-7ded8dd08a9b" containerName="revoke-keys"
Jul 17 21:42:36.925731 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:36.925714    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c36d1fee-cb56-4457-bb7a-7ded8dd08a9b" containerName="revoke-keys"
Jul 17 21:42:36.925880 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:36.925813    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c36d1fee-cb56-4457-bb7a-7ded8dd08a9b" containerName="revoke-keys"
Jul 17 21:42:36.925880 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:36.925830    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c36d1fee-cb56-4457-bb7a-7ded8dd08a9b" containerName="revoke-keys"
Jul 17 21:42:36.930775 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:36.930752    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd"
Jul 17 21:42:36.933730 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:36.933706    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:42:36.933978 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:36.933763    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:42:36.936315 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:36.936256    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd"]
Jul 17 21:42:37.070082 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:37.070046    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd\" (UID: \"8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd"
Jul 17 21:42:37.171117 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:37.171088    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd\" (UID: \"8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd"
Jul 17 21:42:37.171774 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:37.171755    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd\" (UID: \"8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd"
Jul 17 21:42:37.243897 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:37.243802    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd"
Jul 17 21:42:37.390179 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:37.390153    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd"]
Jul 17 21:42:37.392078 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:42:37.392051    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4b1420_9c5e_440a_8180_a0a1ef0c1f3d.slice/crio-ba0cd59dbf20caaccfe6ab98750065eec075d7d7b3a8c3caba620d9debc6a8fa WatchSource:0}: Error finding container ba0cd59dbf20caaccfe6ab98750065eec075d7d7b3a8c3caba620d9debc6a8fa: Status 404 returned error can't find the container with id ba0cd59dbf20caaccfe6ab98750065eec075d7d7b3a8c3caba620d9debc6a8fa
Jul 17 21:42:38.172078 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:38.172046    2569 generic.go:356] "Generic (PLEG): container finished" podID="8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d" containerID="59e9dab86fe1943948a9ae663eb4ef9da8a1bf436338178882115ebe62fcf762" exitCode=7
Jul 17 21:42:38.172496 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:38.172095    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd" event={"ID":"8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d","Type":"ContainerDied","Data":"59e9dab86fe1943948a9ae663eb4ef9da8a1bf436338178882115ebe62fcf762"}
Jul 17 21:42:38.172496 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:38.172137    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd" event={"ID":"8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d","Type":"ContainerStarted","Data":"ba0cd59dbf20caaccfe6ab98750065eec075d7d7b3a8c3caba620d9debc6a8fa"}
Jul 17 21:42:38.172496 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:38.172305    2569 scope.go:117] "RemoveContainer" containerID="59e9dab86fe1943948a9ae663eb4ef9da8a1bf436338178882115ebe62fcf762"
Jul 17 21:42:39.179758 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:39.179722    2569 generic.go:356] "Generic (PLEG): container finished" podID="8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d" containerID="ca2eeef310d51ce7f09b6d57f3fff99abc0e6f1e2143e53ca4d81a28ca458796" exitCode=7
Jul 17 21:42:39.180205 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:39.179776    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd" event={"ID":"8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d","Type":"ContainerDied","Data":"ca2eeef310d51ce7f09b6d57f3fff99abc0e6f1e2143e53ca4d81a28ca458796"}
Jul 17 21:42:39.180205 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:39.179828    2569 scope.go:117] "RemoveContainer" containerID="59e9dab86fe1943948a9ae663eb4ef9da8a1bf436338178882115ebe62fcf762"
Jul 17 21:42:39.180205 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:39.180009    2569 scope.go:117] "RemoveContainer" containerID="ca2eeef310d51ce7f09b6d57f3fff99abc0e6f1e2143e53ca4d81a28ca458796"
Jul 17 21:42:39.180365 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:42:39.180250    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd_odh-ai-gateway-infra(8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd" podUID="8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d"
Jul 17 21:42:40.042334 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:40.042303    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss"]
Jul 17 21:42:40.042749 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:40.042734    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c36d1fee-cb56-4457-bb7a-7ded8dd08a9b" containerName="revoke-keys"
Jul 17 21:42:40.042811 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:40.042751    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c36d1fee-cb56-4457-bb7a-7ded8dd08a9b" containerName="revoke-keys"
Jul 17 21:42:40.042856 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:40.042829    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c36d1fee-cb56-4457-bb7a-7ded8dd08a9b" containerName="revoke-keys"
Jul 17 21:42:40.047077 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:40.047060    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss"
Jul 17 21:42:40.053697 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:40.053673    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss"]
Jul 17 21:42:40.096086 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:40.096056    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3611cd09-aa92-4077-a4ac-226f5b415931-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-b6fss\" (UID: \"3611cd09-aa92-4077-a4ac-226f5b415931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss"
Jul 17 21:42:40.187221 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:40.187193    2569 scope.go:117] "RemoveContainer" containerID="ca2eeef310d51ce7f09b6d57f3fff99abc0e6f1e2143e53ca4d81a28ca458796"
Jul 17 21:42:40.187624 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:42:40.187427    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd_odh-ai-gateway-infra(8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd" podUID="8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d"
Jul 17 21:42:40.197560 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:40.197534    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3611cd09-aa92-4077-a4ac-226f5b415931-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-b6fss\" (UID: \"3611cd09-aa92-4077-a4ac-226f5b415931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss"
Jul 17 21:42:40.198098 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:40.198082    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3611cd09-aa92-4077-a4ac-226f5b415931-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-b6fss\" (UID: \"3611cd09-aa92-4077-a4ac-226f5b415931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss"
Jul 17 21:42:40.358752 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:40.358720    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss"
Jul 17 21:42:40.448259 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:40.448226    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:42:40.496031 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:40.496004    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss"]
Jul 17 21:42:40.498593 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:42:40.498567    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3611cd09_aa92_4077_a4ac_226f5b415931.slice/crio-5819c880e96cbbb3e8ed1d91aa9b10dac567d976c8ab89c4208e415ef8a0933b WatchSource:0}: Error finding container 5819c880e96cbbb3e8ed1d91aa9b10dac567d976c8ab89c4208e415ef8a0933b: Status 404 returned error can't find the container with id 5819c880e96cbbb3e8ed1d91aa9b10dac567d976c8ab89c4208e415ef8a0933b
Jul 17 21:42:41.193731 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:41.193691    2569 generic.go:356] "Generic (PLEG): container finished" podID="3611cd09-aa92-4077-a4ac-226f5b415931" containerID="a7a90490f82a0a4dfdf6705379471c4ca42bcbb5eec895ba0f4a1e00bc320135" exitCode=6
Jul 17 21:42:41.194126 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:41.193777    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss" event={"ID":"3611cd09-aa92-4077-a4ac-226f5b415931","Type":"ContainerDied","Data":"a7a90490f82a0a4dfdf6705379471c4ca42bcbb5eec895ba0f4a1e00bc320135"}
Jul 17 21:42:41.194126 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:41.193838    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss" event={"ID":"3611cd09-aa92-4077-a4ac-226f5b415931","Type":"ContainerStarted","Data":"5819c880e96cbbb3e8ed1d91aa9b10dac567d976c8ab89c4208e415ef8a0933b"}
Jul 17 21:42:41.194126 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:41.193972    2569 scope.go:117] "RemoveContainer" containerID="a7a90490f82a0a4dfdf6705379471c4ca42bcbb5eec895ba0f4a1e00bc320135"
Jul 17 21:42:42.201079 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:42.200996    2569 generic.go:356] "Generic (PLEG): container finished" podID="3611cd09-aa92-4077-a4ac-226f5b415931" containerID="90047eadb1b0181b8318610deed4f020808d8608abf25f4395de110f7dc3c507" exitCode=6
Jul 17 21:42:42.201569 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:42.201078    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss" event={"ID":"3611cd09-aa92-4077-a4ac-226f5b415931","Type":"ContainerDied","Data":"90047eadb1b0181b8318610deed4f020808d8608abf25f4395de110f7dc3c507"}
Jul 17 21:42:42.201569 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:42.201148    2569 scope.go:117] "RemoveContainer" containerID="a7a90490f82a0a4dfdf6705379471c4ca42bcbb5eec895ba0f4a1e00bc320135"
Jul 17 21:42:42.201569 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:42.201259    2569 scope.go:117] "RemoveContainer" containerID="90047eadb1b0181b8318610deed4f020808d8608abf25f4395de110f7dc3c507"
Jul 17 21:42:42.201569 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:42:42.201530    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-b6fss_odh-ai-gateway-infra(3611cd09-aa92-4077-a4ac-226f5b415931)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss" podUID="3611cd09-aa92-4077-a4ac-226f5b415931"
Jul 17 21:42:43.208314 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:43.208286    2569 scope.go:117] "RemoveContainer" containerID="90047eadb1b0181b8318610deed4f020808d8608abf25f4395de110f7dc3c507"
Jul 17 21:42:43.208767 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:42:43.208505    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-b6fss_odh-ai-gateway-infra(3611cd09-aa92-4077-a4ac-226f5b415931)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss" podUID="3611cd09-aa92-4077-a4ac-226f5b415931"
Jul 17 21:42:54.651185 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:54.651148    2569 scope.go:117] "RemoveContainer" containerID="ca2eeef310d51ce7f09b6d57f3fff99abc0e6f1e2143e53ca4d81a28ca458796"
Jul 17 21:42:55.278983 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:55.278953    2569 generic.go:356] "Generic (PLEG): container finished" podID="8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d" containerID="23fd584cc08f0598190cc8649539af36eb5cbb8c1507e976598eff95149a2d6c" exitCode=7
Jul 17 21:42:55.279141 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:55.279024    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd" event={"ID":"8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d","Type":"ContainerDied","Data":"23fd584cc08f0598190cc8649539af36eb5cbb8c1507e976598eff95149a2d6c"}
Jul 17 21:42:55.279141 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:55.279087    2569 scope.go:117] "RemoveContainer" containerID="ca2eeef310d51ce7f09b6d57f3fff99abc0e6f1e2143e53ca4d81a28ca458796"
Jul 17 21:42:55.279381 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:55.279367    2569 scope.go:117] "RemoveContainer" containerID="23fd584cc08f0598190cc8649539af36eb5cbb8c1507e976598eff95149a2d6c"
Jul 17 21:42:55.279677 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:42:55.279646    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd_odh-ai-gateway-infra(8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd" podUID="8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d"
Jul 17 21:42:55.650385 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:55.650350    2569 scope.go:117] "RemoveContainer" containerID="90047eadb1b0181b8318610deed4f020808d8608abf25f4395de110f7dc3c507"
Jul 17 21:42:56.285962 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:56.285930    2569 generic.go:356] "Generic (PLEG): container finished" podID="3611cd09-aa92-4077-a4ac-226f5b415931" containerID="8819de10daf021fd69c6962d642f9e42f7c6ebc4fbfbf9f3677a76c81bbbe2c9" exitCode=6
Jul 17 21:42:56.286340 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:56.286000    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss" event={"ID":"3611cd09-aa92-4077-a4ac-226f5b415931","Type":"ContainerDied","Data":"8819de10daf021fd69c6962d642f9e42f7c6ebc4fbfbf9f3677a76c81bbbe2c9"}
Jul 17 21:42:56.286340 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:56.286058    2569 scope.go:117] "RemoveContainer" containerID="90047eadb1b0181b8318610deed4f020808d8608abf25f4395de110f7dc3c507"
Jul 17 21:42:56.286495 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:56.286343    2569 scope.go:117] "RemoveContainer" containerID="8819de10daf021fd69c6962d642f9e42f7c6ebc4fbfbf9f3677a76c81bbbe2c9"
Jul 17 21:42:56.286634 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:42:56.286610    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-b6fss_odh-ai-gateway-infra(3611cd09-aa92-4077-a4ac-226f5b415931)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss" podUID="3611cd09-aa92-4077-a4ac-226f5b415931"
Jul 17 21:42:56.305437 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:56.305395    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd"]
Jul 17 21:42:56.448729 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:56.448703    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd"
Jul 17 21:42:56.530272 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:56.530178    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d-maas-api-service-ca\") pod \"8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d\" (UID: \"8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d\") "
Jul 17 21:42:56.530569 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:56.530545    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d" (UID: "8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:42:56.630980 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:56.630940    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:42:57.295285 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:57.295245    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd" event={"ID":"8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d","Type":"ContainerDied","Data":"ba0cd59dbf20caaccfe6ab98750065eec075d7d7b3a8c3caba620d9debc6a8fa"}
Jul 17 21:42:57.295285 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:57.295272    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd"
Jul 17 21:42:57.295750 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:57.295316    2569 scope.go:117] "RemoveContainer" containerID="23fd584cc08f0598190cc8649539af36eb5cbb8c1507e976598eff95149a2d6c"
Jul 17 21:42:57.310323 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:57.310287    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss"]
Jul 17 21:42:57.313295 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:57.313268    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd"]
Jul 17 21:42:57.320333 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:57.320307    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-2fkkd"]
Jul 17 21:42:57.453275 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:57.453243    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss"
Jul 17 21:42:57.538540 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:57.538509    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3611cd09-aa92-4077-a4ac-226f5b415931-maas-api-service-ca\") pod \"3611cd09-aa92-4077-a4ac-226f5b415931\" (UID: \"3611cd09-aa92-4077-a4ac-226f5b415931\") "
Jul 17 21:42:57.538883 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:57.538862    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3611cd09-aa92-4077-a4ac-226f5b415931-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3611cd09-aa92-4077-a4ac-226f5b415931" (UID: "3611cd09-aa92-4077-a4ac-226f5b415931"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:42:57.639653 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:57.639621    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3611cd09-aa92-4077-a4ac-226f5b415931-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:42:58.303213 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.303181    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss" event={"ID":"3611cd09-aa92-4077-a4ac-226f5b415931","Type":"ContainerDied","Data":"5819c880e96cbbb3e8ed1d91aa9b10dac567d976c8ab89c4208e415ef8a0933b"}
Jul 17 21:42:58.303735 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.303216    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss"
Jul 17 21:42:58.303735 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.303250    2569 scope.go:117] "RemoveContainer" containerID="8819de10daf021fd69c6962d642f9e42f7c6ebc4fbfbf9f3677a76c81bbbe2c9"
Jul 17 21:42:58.327737 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.327704    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss"]
Jul 17 21:42:58.331488 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.331457    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b6fss"]
Jul 17 21:42:58.656258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.656221    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3611cd09-aa92-4077-a4ac-226f5b415931" path="/var/lib/kubelet/pods/3611cd09-aa92-4077-a4ac-226f5b415931/volumes"
Jul 17 21:42:58.656605 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.656589    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d" path="/var/lib/kubelet/pods/8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d/volumes"
Jul 17 21:42:58.868893 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.868859    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4"]
Jul 17 21:42:58.869255 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.869241    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d" containerName="revoke-keys"
Jul 17 21:42:58.869317 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.869258    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d" containerName="revoke-keys"
Jul 17 21:42:58.869317 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.869280    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d" containerName="revoke-keys"
Jul 17 21:42:58.869317 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.869288    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d" containerName="revoke-keys"
Jul 17 21:42:58.869317 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.869305    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3611cd09-aa92-4077-a4ac-226f5b415931" containerName="revoke-keys"
Jul 17 21:42:58.869317 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.869316    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3611cd09-aa92-4077-a4ac-226f5b415931" containerName="revoke-keys"
Jul 17 21:42:58.869537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.869343    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3611cd09-aa92-4077-a4ac-226f5b415931" containerName="revoke-keys"
Jul 17 21:42:58.869537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.869350    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3611cd09-aa92-4077-a4ac-226f5b415931" containerName="revoke-keys"
Jul 17 21:42:58.869537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.869449    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d" containerName="revoke-keys"
Jul 17 21:42:58.869537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.869460    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3611cd09-aa92-4077-a4ac-226f5b415931" containerName="revoke-keys"
Jul 17 21:42:58.869537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.869468    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3611cd09-aa92-4077-a4ac-226f5b415931" containerName="revoke-keys"
Jul 17 21:42:58.869537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.869476    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d" containerName="revoke-keys"
Jul 17 21:42:58.873748 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.873731    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4"
Jul 17 21:42:58.875775 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.875753    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:42:58.875933 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.875826    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:42:58.879901 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.879877    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4"]
Jul 17 21:42:58.951926 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:58.951825    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ccb6d21-458e-4994-b071-3b583eee810c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4\" (UID: \"6ccb6d21-458e-4994-b071-3b583eee810c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4"
Jul 17 21:42:59.053179 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:59.053148    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ccb6d21-458e-4994-b071-3b583eee810c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4\" (UID: \"6ccb6d21-458e-4994-b071-3b583eee810c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4"
Jul 17 21:42:59.053850 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:59.053829    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ccb6d21-458e-4994-b071-3b583eee810c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4\" (UID: \"6ccb6d21-458e-4994-b071-3b583eee810c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4"
Jul 17 21:42:59.185906 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:59.185866    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4"
Jul 17 21:42:59.330142 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:42:59.324742    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4"]
Jul 17 21:43:00.319943 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:00.319856    2569 generic.go:356] "Generic (PLEG): container finished" podID="6ccb6d21-458e-4994-b071-3b583eee810c" containerID="1d24b3183f1a5551b434892a01dc194de50066aa2fa9394d5851d8f3abfcdfde" exitCode=6
Jul 17 21:43:00.320086 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:00.319946    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4" event={"ID":"6ccb6d21-458e-4994-b071-3b583eee810c","Type":"ContainerDied","Data":"1d24b3183f1a5551b434892a01dc194de50066aa2fa9394d5851d8f3abfcdfde"}
Jul 17 21:43:00.320086 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:00.319999    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4" event={"ID":"6ccb6d21-458e-4994-b071-3b583eee810c","Type":"ContainerStarted","Data":"52d01af395963bbf979f45f7b1df0ddff49393b456ab140629c79b75c58b10ef"}
Jul 17 21:43:00.320200 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:00.320166    2569 scope.go:117] "RemoveContainer" containerID="1d24b3183f1a5551b434892a01dc194de50066aa2fa9394d5851d8f3abfcdfde"
Jul 17 21:43:01.327545 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:01.327459    2569 generic.go:356] "Generic (PLEG): container finished" podID="6ccb6d21-458e-4994-b071-3b583eee810c" containerID="5aeac3999580ee5a5e3b2da08c95b01adfabbab7b10234e049e3b1ba5897cdc0" exitCode=6
Jul 17 21:43:01.328011 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:01.327543    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4" event={"ID":"6ccb6d21-458e-4994-b071-3b583eee810c","Type":"ContainerDied","Data":"5aeac3999580ee5a5e3b2da08c95b01adfabbab7b10234e049e3b1ba5897cdc0"}
Jul 17 21:43:01.328011 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:01.327611    2569 scope.go:117] "RemoveContainer" containerID="1d24b3183f1a5551b434892a01dc194de50066aa2fa9394d5851d8f3abfcdfde"
Jul 17 21:43:01.328011 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:01.327716    2569 scope.go:117] "RemoveContainer" containerID="5aeac3999580ee5a5e3b2da08c95b01adfabbab7b10234e049e3b1ba5897cdc0"
Jul 17 21:43:01.328011 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:43:01.327939    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4_odh-ai-gateway-infra(6ccb6d21-458e-4994-b071-3b583eee810c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4" podUID="6ccb6d21-458e-4994-b071-3b583eee810c"
Jul 17 21:43:02.334972 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:02.334938    2569 scope.go:117] "RemoveContainer" containerID="5aeac3999580ee5a5e3b2da08c95b01adfabbab7b10234e049e3b1ba5897cdc0"
Jul 17 21:43:02.335362 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:43:02.335144    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4_odh-ai-gateway-infra(6ccb6d21-458e-4994-b071-3b583eee810c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4" podUID="6ccb6d21-458e-4994-b071-3b583eee810c"
Jul 17 21:43:06.788907 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:06.788874    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:43:06.812728 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:06.812704    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:43:17.651040 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:17.651004    2569 scope.go:117] "RemoveContainer" containerID="5aeac3999580ee5a5e3b2da08c95b01adfabbab7b10234e049e3b1ba5897cdc0"
Jul 17 21:43:18.427525 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:18.427490    2569 generic.go:356] "Generic (PLEG): container finished" podID="6ccb6d21-458e-4994-b071-3b583eee810c" containerID="6ff61f1012e3c774ebccfa0d954dafe240a668971642af6bb2bcbde62771783a" exitCode=6
Jul 17 21:43:18.427736 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:18.427561    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4" event={"ID":"6ccb6d21-458e-4994-b071-3b583eee810c","Type":"ContainerDied","Data":"6ff61f1012e3c774ebccfa0d954dafe240a668971642af6bb2bcbde62771783a"}
Jul 17 21:43:18.427736 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:18.427636    2569 scope.go:117] "RemoveContainer" containerID="5aeac3999580ee5a5e3b2da08c95b01adfabbab7b10234e049e3b1ba5897cdc0"
Jul 17 21:43:18.427944 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:18.427928    2569 scope.go:117] "RemoveContainer" containerID="6ff61f1012e3c774ebccfa0d954dafe240a668971642af6bb2bcbde62771783a"
Jul 17 21:43:18.428165 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:43:18.428146    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4_odh-ai-gateway-infra(6ccb6d21-458e-4994-b071-3b583eee810c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4" podUID="6ccb6d21-458e-4994-b071-3b583eee810c"
Jul 17 21:43:19.452947 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:19.452910    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4"]
Jul 17 21:43:19.609135 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:19.609107    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4"
Jul 17 21:43:19.732085 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:19.732004    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ccb6d21-458e-4994-b071-3b583eee810c-maas-api-service-ca\") pod \"6ccb6d21-458e-4994-b071-3b583eee810c\" (UID: \"6ccb6d21-458e-4994-b071-3b583eee810c\") "
Jul 17 21:43:19.732370 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:19.732348    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ccb6d21-458e-4994-b071-3b583eee810c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ccb6d21-458e-4994-b071-3b583eee810c" (UID: "6ccb6d21-458e-4994-b071-3b583eee810c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:43:19.833560 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:19.833523    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ccb6d21-458e-4994-b071-3b583eee810c-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:43:20.442562 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:20.442530    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4"
Jul 17 21:43:20.442748 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:20.442576    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4" event={"ID":"6ccb6d21-458e-4994-b071-3b583eee810c","Type":"ContainerDied","Data":"52d01af395963bbf979f45f7b1df0ddff49393b456ab140629c79b75c58b10ef"}
Jul 17 21:43:20.442748 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:20.442644    2569 scope.go:117] "RemoveContainer" containerID="6ff61f1012e3c774ebccfa0d954dafe240a668971642af6bb2bcbde62771783a"
Jul 17 21:43:20.465454 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:20.465419    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4"]
Jul 17 21:43:20.471418 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:20.471367    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-f45t4"]
Jul 17 21:43:20.657054 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:20.657008    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ccb6d21-458e-4994-b071-3b583eee810c" path="/var/lib/kubelet/pods/6ccb6d21-458e-4994-b071-3b583eee810c/volumes"
Jul 17 21:43:30.090452 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.090410    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh"]
Jul 17 21:43:30.090890 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.090850    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ccb6d21-458e-4994-b071-3b583eee810c" containerName="revoke-keys"
Jul 17 21:43:30.090890 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.090862    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ccb6d21-458e-4994-b071-3b583eee810c" containerName="revoke-keys"
Jul 17 21:43:30.090890 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.090871    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ccb6d21-458e-4994-b071-3b583eee810c" containerName="revoke-keys"
Jul 17 21:43:30.090890 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.090880    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ccb6d21-458e-4994-b071-3b583eee810c" containerName="revoke-keys"
Jul 17 21:43:30.091050 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.090894    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ccb6d21-458e-4994-b071-3b583eee810c" containerName="revoke-keys"
Jul 17 21:43:30.091050 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.090904    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ccb6d21-458e-4994-b071-3b583eee810c" containerName="revoke-keys"
Jul 17 21:43:30.091050 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.090918    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d" containerName="revoke-keys"
Jul 17 21:43:30.091050 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.090925    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d" containerName="revoke-keys"
Jul 17 21:43:30.091050 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.090932    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3611cd09-aa92-4077-a4ac-226f5b415931" containerName="revoke-keys"
Jul 17 21:43:30.091050 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.090937    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3611cd09-aa92-4077-a4ac-226f5b415931" containerName="revoke-keys"
Jul 17 21:43:30.091050 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.091005    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c4b1420-9c5e-440a-8180-a0a1ef0c1f3d" containerName="revoke-keys"
Jul 17 21:43:30.091050 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.091012    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3611cd09-aa92-4077-a4ac-226f5b415931" containerName="revoke-keys"
Jul 17 21:43:30.091050 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.091017    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ccb6d21-458e-4994-b071-3b583eee810c" containerName="revoke-keys"
Jul 17 21:43:30.091050 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.091023    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ccb6d21-458e-4994-b071-3b583eee810c" containerName="revoke-keys"
Jul 17 21:43:30.091050 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.091029    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ccb6d21-458e-4994-b071-3b583eee810c" containerName="revoke-keys"
Jul 17 21:43:30.095628 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.095603    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh"
Jul 17 21:43:30.097578 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.097556    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:43:30.097708 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.097600    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:43:30.103005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.102851    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh"]
Jul 17 21:43:30.236908 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.236861    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c3bbfd-7b8e-4b8f-848c-e38acf68a38c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-xvlrh\" (UID: \"60c3bbfd-7b8e-4b8f-848c-e38acf68a38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh"
Jul 17 21:43:30.337791 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.337753    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c3bbfd-7b8e-4b8f-848c-e38acf68a38c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-xvlrh\" (UID: \"60c3bbfd-7b8e-4b8f-848c-e38acf68a38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh"
Jul 17 21:43:30.338370 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.338347    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c3bbfd-7b8e-4b8f-848c-e38acf68a38c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-xvlrh\" (UID: \"60c3bbfd-7b8e-4b8f-848c-e38acf68a38c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh"
Jul 17 21:43:30.408215 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.408116    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh"
Jul 17 21:43:30.548710 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:30.548636    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh"]
Jul 17 21:43:30.551231 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:43:30.551202    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c3bbfd_7b8e_4b8f_848c_e38acf68a38c.slice/crio-d3ce7932dc7e057cd8d10e71d723eef70aa14f40b9198b6e71f1d7502f200bf9 WatchSource:0}: Error finding container d3ce7932dc7e057cd8d10e71d723eef70aa14f40b9198b6e71f1d7502f200bf9: Status 404 returned error can't find the container with id d3ce7932dc7e057cd8d10e71d723eef70aa14f40b9198b6e71f1d7502f200bf9
Jul 17 21:43:31.513372 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:31.513271    2569 generic.go:356] "Generic (PLEG): container finished" podID="60c3bbfd-7b8e-4b8f-848c-e38acf68a38c" containerID="4806f22a40a949f09334bb3367bbc86be6e658bed94ab68716625c1f8f451795" exitCode=6
Jul 17 21:43:31.513372 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:31.513347    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh" event={"ID":"60c3bbfd-7b8e-4b8f-848c-e38acf68a38c","Type":"ContainerDied","Data":"4806f22a40a949f09334bb3367bbc86be6e658bed94ab68716625c1f8f451795"}
Jul 17 21:43:31.513875 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:31.513395    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh" event={"ID":"60c3bbfd-7b8e-4b8f-848c-e38acf68a38c","Type":"ContainerStarted","Data":"d3ce7932dc7e057cd8d10e71d723eef70aa14f40b9198b6e71f1d7502f200bf9"}
Jul 17 21:43:31.513875 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:31.513640    2569 scope.go:117] "RemoveContainer" containerID="4806f22a40a949f09334bb3367bbc86be6e658bed94ab68716625c1f8f451795"
Jul 17 21:43:32.521158 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:32.521071    2569 generic.go:356] "Generic (PLEG): container finished" podID="60c3bbfd-7b8e-4b8f-848c-e38acf68a38c" containerID="7a00d2799d0421b3796862205540e2031416b2849e1a8d58954880c58d957993" exitCode=6
Jul 17 21:43:32.521158 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:32.521127    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh" event={"ID":"60c3bbfd-7b8e-4b8f-848c-e38acf68a38c","Type":"ContainerDied","Data":"7a00d2799d0421b3796862205540e2031416b2849e1a8d58954880c58d957993"}
Jul 17 21:43:32.521658 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:32.521178    2569 scope.go:117] "RemoveContainer" containerID="4806f22a40a949f09334bb3367bbc86be6e658bed94ab68716625c1f8f451795"
Jul 17 21:43:32.521658 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:32.521354    2569 scope.go:117] "RemoveContainer" containerID="7a00d2799d0421b3796862205540e2031416b2849e1a8d58954880c58d957993"
Jul 17 21:43:32.521658 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:43:32.521625    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-xvlrh_odh-ai-gateway-infra(60c3bbfd-7b8e-4b8f-848c-e38acf68a38c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh" podUID="60c3bbfd-7b8e-4b8f-848c-e38acf68a38c"
Jul 17 21:43:33.528090 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:33.528056    2569 scope.go:117] "RemoveContainer" containerID="7a00d2799d0421b3796862205540e2031416b2849e1a8d58954880c58d957993"
Jul 17 21:43:33.528581 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:43:33.528343    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-xvlrh_odh-ai-gateway-infra(60c3bbfd-7b8e-4b8f-848c-e38acf68a38c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh" podUID="60c3bbfd-7b8e-4b8f-848c-e38acf68a38c"
Jul 17 21:43:37.027431 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:37.027383    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t"]
Jul 17 21:43:37.032104 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:37.032085    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t"
Jul 17 21:43:37.039321 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:37.039280    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t"]
Jul 17 21:43:37.098837 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:37.098791    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd83f4e-5cd8-48d9-9b77-34627986c46d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t\" (UID: \"dfd83f4e-5cd8-48d9-9b77-34627986c46d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t"
Jul 17 21:43:37.199431 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:37.199365    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd83f4e-5cd8-48d9-9b77-34627986c46d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t\" (UID: \"dfd83f4e-5cd8-48d9-9b77-34627986c46d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t"
Jul 17 21:43:37.200013 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:37.199993    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd83f4e-5cd8-48d9-9b77-34627986c46d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t\" (UID: \"dfd83f4e-5cd8-48d9-9b77-34627986c46d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t"
Jul 17 21:43:37.344800 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:37.344752    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t"
Jul 17 21:43:37.483784 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:37.483754    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t"]
Jul 17 21:43:37.486134 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:43:37.486106    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd83f4e_5cd8_48d9_9b77_34627986c46d.slice/crio-63d610010d6db91b8acc0f5853d94e01e236eb8662e4119b055649d7d1c1ba51 WatchSource:0}: Error finding container 63d610010d6db91b8acc0f5853d94e01e236eb8662e4119b055649d7d1c1ba51: Status 404 returned error can't find the container with id 63d610010d6db91b8acc0f5853d94e01e236eb8662e4119b055649d7d1c1ba51
Jul 17 21:43:37.551588 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:37.551551    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t" event={"ID":"dfd83f4e-5cd8-48d9-9b77-34627986c46d","Type":"ContainerStarted","Data":"63d610010d6db91b8acc0f5853d94e01e236eb8662e4119b055649d7d1c1ba51"}
Jul 17 21:43:38.559392 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:38.559291    2569 generic.go:356] "Generic (PLEG): container finished" podID="dfd83f4e-5cd8-48d9-9b77-34627986c46d" containerID="167af2e26272743824e83cf4cec19396428672f9817cd284819068dc5eaa7ebf" exitCode=7
Jul 17 21:43:38.559392 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:38.559378    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t" event={"ID":"dfd83f4e-5cd8-48d9-9b77-34627986c46d","Type":"ContainerDied","Data":"167af2e26272743824e83cf4cec19396428672f9817cd284819068dc5eaa7ebf"}
Jul 17 21:43:38.559837 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:38.559569    2569 scope.go:117] "RemoveContainer" containerID="167af2e26272743824e83cf4cec19396428672f9817cd284819068dc5eaa7ebf"
Jul 17 21:43:39.567285 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:39.567193    2569 generic.go:356] "Generic (PLEG): container finished" podID="dfd83f4e-5cd8-48d9-9b77-34627986c46d" containerID="86b7bea7e8d9f4825850a0b8cc45533bf4ed2c857b703665dd9957cf29f8dcce" exitCode=7
Jul 17 21:43:39.567698 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:39.567276    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t" event={"ID":"dfd83f4e-5cd8-48d9-9b77-34627986c46d","Type":"ContainerDied","Data":"86b7bea7e8d9f4825850a0b8cc45533bf4ed2c857b703665dd9957cf29f8dcce"}
Jul 17 21:43:39.567698 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:39.567337    2569 scope.go:117] "RemoveContainer" containerID="167af2e26272743824e83cf4cec19396428672f9817cd284819068dc5eaa7ebf"
Jul 17 21:43:39.567698 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:39.567635    2569 scope.go:117] "RemoveContainer" containerID="86b7bea7e8d9f4825850a0b8cc45533bf4ed2c857b703665dd9957cf29f8dcce"
Jul 17 21:43:39.567881 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:43:39.567860    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t_odh-ai-gateway-infra(dfd83f4e-5cd8-48d9-9b77-34627986c46d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t" podUID="dfd83f4e-5cd8-48d9-9b77-34627986c46d"
Jul 17 21:43:40.575119 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:40.575090    2569 scope.go:117] "RemoveContainer" containerID="86b7bea7e8d9f4825850a0b8cc45533bf4ed2c857b703665dd9957cf29f8dcce"
Jul 17 21:43:40.575525 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:43:40.575262    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t_odh-ai-gateway-infra(dfd83f4e-5cd8-48d9-9b77-34627986c46d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t" podUID="dfd83f4e-5cd8-48d9-9b77-34627986c46d"
Jul 17 21:43:47.650453 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:47.650389    2569 scope.go:117] "RemoveContainer" containerID="7a00d2799d0421b3796862205540e2031416b2849e1a8d58954880c58d957993"
Jul 17 21:43:48.622273 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:48.622240    2569 generic.go:356] "Generic (PLEG): container finished" podID="60c3bbfd-7b8e-4b8f-848c-e38acf68a38c" containerID="f36d2c2a9371db2cab37b344f7c269bacb6d2502fa5686dfdffc2ed6fb49949a" exitCode=6
Jul 17 21:43:48.622469 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:48.622316    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh" event={"ID":"60c3bbfd-7b8e-4b8f-848c-e38acf68a38c","Type":"ContainerDied","Data":"f36d2c2a9371db2cab37b344f7c269bacb6d2502fa5686dfdffc2ed6fb49949a"}
Jul 17 21:43:48.622469 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:48.622384    2569 scope.go:117] "RemoveContainer" containerID="7a00d2799d0421b3796862205540e2031416b2849e1a8d58954880c58d957993"
Jul 17 21:43:48.622726 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:48.622708    2569 scope.go:117] "RemoveContainer" containerID="f36d2c2a9371db2cab37b344f7c269bacb6d2502fa5686dfdffc2ed6fb49949a"
Jul 17 21:43:48.622980 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:43:48.622960    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-xvlrh_odh-ai-gateway-infra(60c3bbfd-7b8e-4b8f-848c-e38acf68a38c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh" podUID="60c3bbfd-7b8e-4b8f-848c-e38acf68a38c"
Jul 17 21:43:49.648376 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:49.648334    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh"]
Jul 17 21:43:49.790268 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:49.790243    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh"
Jul 17 21:43:49.901500 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:49.901383    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c3bbfd-7b8e-4b8f-848c-e38acf68a38c-maas-api-service-ca\") pod \"60c3bbfd-7b8e-4b8f-848c-e38acf68a38c\" (UID: \"60c3bbfd-7b8e-4b8f-848c-e38acf68a38c\") "
Jul 17 21:43:49.901751 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:49.901726    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c3bbfd-7b8e-4b8f-848c-e38acf68a38c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60c3bbfd-7b8e-4b8f-848c-e38acf68a38c" (UID: "60c3bbfd-7b8e-4b8f-848c-e38acf68a38c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:43:50.002457 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:50.002389    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c3bbfd-7b8e-4b8f-848c-e38acf68a38c-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:43:50.635956 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:50.635923    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh"
Jul 17 21:43:50.635956 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:50.635950    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh" event={"ID":"60c3bbfd-7b8e-4b8f-848c-e38acf68a38c","Type":"ContainerDied","Data":"d3ce7932dc7e057cd8d10e71d723eef70aa14f40b9198b6e71f1d7502f200bf9"}
Jul 17 21:43:50.636191 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:50.636049    2569 scope.go:117] "RemoveContainer" containerID="f36d2c2a9371db2cab37b344f7c269bacb6d2502fa5686dfdffc2ed6fb49949a"
Jul 17 21:43:50.658827 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:50.658797    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh"]
Jul 17 21:43:50.667087 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:50.667066    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-xvlrh"]
Jul 17 21:43:51.650937 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:51.650910    2569 scope.go:117] "RemoveContainer" containerID="86b7bea7e8d9f4825850a0b8cc45533bf4ed2c857b703665dd9957cf29f8dcce"
Jul 17 21:43:52.653553 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:52.653455    2569 generic.go:356] "Generic (PLEG): container finished" podID="dfd83f4e-5cd8-48d9-9b77-34627986c46d" containerID="bb0bf5e4f45d2522a3aefa2053184dbbd885c3291cfaea74da7f00c4a0b30850" exitCode=7
Jul 17 21:43:52.656298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:52.656272    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60c3bbfd-7b8e-4b8f-848c-e38acf68a38c" path="/var/lib/kubelet/pods/60c3bbfd-7b8e-4b8f-848c-e38acf68a38c/volumes"
Jul 17 21:43:52.656595 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:52.656577    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t" event={"ID":"dfd83f4e-5cd8-48d9-9b77-34627986c46d","Type":"ContainerDied","Data":"bb0bf5e4f45d2522a3aefa2053184dbbd885c3291cfaea74da7f00c4a0b30850"}
Jul 17 21:43:52.656665 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:52.656631    2569 scope.go:117] "RemoveContainer" containerID="86b7bea7e8d9f4825850a0b8cc45533bf4ed2c857b703665dd9957cf29f8dcce"
Jul 17 21:43:52.656941 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:52.656916    2569 scope.go:117] "RemoveContainer" containerID="bb0bf5e4f45d2522a3aefa2053184dbbd885c3291cfaea74da7f00c4a0b30850"
Jul 17 21:43:52.657175 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:43:52.657151    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t_odh-ai-gateway-infra(dfd83f4e-5cd8-48d9-9b77-34627986c46d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t" podUID="dfd83f4e-5cd8-48d9-9b77-34627986c46d"
Jul 17 21:43:53.685233 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:53.685202    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t"]
Jul 17 21:43:53.822264 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:53.822239    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t"
Jul 17 21:43:53.936714 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:53.936616    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd83f4e-5cd8-48d9-9b77-34627986c46d-maas-api-service-ca\") pod \"dfd83f4e-5cd8-48d9-9b77-34627986c46d\" (UID: \"dfd83f4e-5cd8-48d9-9b77-34627986c46d\") "
Jul 17 21:43:53.937045 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:53.937015    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd83f4e-5cd8-48d9-9b77-34627986c46d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dfd83f4e-5cd8-48d9-9b77-34627986c46d" (UID: "dfd83f4e-5cd8-48d9-9b77-34627986c46d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:43:54.037498 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:54.037457    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfd83f4e-5cd8-48d9-9b77-34627986c46d-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:43:54.667085 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:54.667063    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t"
Jul 17 21:43:54.667209 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:54.667104    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t" event={"ID":"dfd83f4e-5cd8-48d9-9b77-34627986c46d","Type":"ContainerDied","Data":"63d610010d6db91b8acc0f5853d94e01e236eb8662e4119b055649d7d1c1ba51"}
Jul 17 21:43:54.667209 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:54.667169    2569 scope.go:117] "RemoveContainer" containerID="bb0bf5e4f45d2522a3aefa2053184dbbd885c3291cfaea74da7f00c4a0b30850"
Jul 17 21:43:54.688975 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:54.688924    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t"]
Jul 17 21:43:54.689865 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:54.689841    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-cpv6t"]
Jul 17 21:43:56.656719 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:56.656685    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfd83f4e-5cd8-48d9-9b77-34627986c46d" path="/var/lib/kubelet/pods/dfd83f4e-5cd8-48d9-9b77-34627986c46d/volumes"
Jul 17 21:43:58.923723 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.923685    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp"]
Jul 17 21:43:58.924111 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.924095    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c3bbfd-7b8e-4b8f-848c-e38acf68a38c" containerName="revoke-keys"
Jul 17 21:43:58.924111 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.924107    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c3bbfd-7b8e-4b8f-848c-e38acf68a38c" containerName="revoke-keys"
Jul 17 21:43:58.924190 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.924179    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c3bbfd-7b8e-4b8f-848c-e38acf68a38c" containerName="revoke-keys"
Jul 17 21:43:58.924190 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.924190    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c3bbfd-7b8e-4b8f-848c-e38acf68a38c" containerName="revoke-keys"
Jul 17 21:43:58.924260 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.924207    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfd83f4e-5cd8-48d9-9b77-34627986c46d" containerName="revoke-keys"
Jul 17 21:43:58.924260 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.924213    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfd83f4e-5cd8-48d9-9b77-34627986c46d" containerName="revoke-keys"
Jul 17 21:43:58.924260 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.924221    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfd83f4e-5cd8-48d9-9b77-34627986c46d" containerName="revoke-keys"
Jul 17 21:43:58.924260 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.924228    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfd83f4e-5cd8-48d9-9b77-34627986c46d" containerName="revoke-keys"
Jul 17 21:43:58.924260 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.924241    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfd83f4e-5cd8-48d9-9b77-34627986c46d" containerName="revoke-keys"
Jul 17 21:43:58.924260 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.924248    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfd83f4e-5cd8-48d9-9b77-34627986c46d" containerName="revoke-keys"
Jul 17 21:43:58.924481 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.924328    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c3bbfd-7b8e-4b8f-848c-e38acf68a38c" containerName="revoke-keys"
Jul 17 21:43:58.924481 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.924337    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c3bbfd-7b8e-4b8f-848c-e38acf68a38c" containerName="revoke-keys"
Jul 17 21:43:58.924481 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.924343    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfd83f4e-5cd8-48d9-9b77-34627986c46d" containerName="revoke-keys"
Jul 17 21:43:58.924481 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.924349    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfd83f4e-5cd8-48d9-9b77-34627986c46d" containerName="revoke-keys"
Jul 17 21:43:58.924481 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.924356    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfd83f4e-5cd8-48d9-9b77-34627986c46d" containerName="revoke-keys"
Jul 17 21:43:58.928920 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.928901    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp"
Jul 17 21:43:58.931116 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.931088    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:43:58.931258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.931195    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:43:58.933848 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.933815    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp"]
Jul 17 21:43:58.981106 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:58.981066    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp\" (UID: \"83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp"
Jul 17 21:43:59.082015 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:59.081972    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp\" (UID: \"83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp"
Jul 17 21:43:59.082622 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:59.082595    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp\" (UID: \"83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp"
Jul 17 21:43:59.241395 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:59.241306    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp"
Jul 17 21:43:59.378235 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:59.378208    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp"]
Jul 17 21:43:59.379586 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:43:59.379552    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83cc5d8a_a0f7_4c7a_9e3b_412c9e36a588.slice/crio-d8c7d9cb616f4bb56429a1dd9bd232b960e0245b470000e4870f3dcd23de5e54 WatchSource:0}: Error finding container d8c7d9cb616f4bb56429a1dd9bd232b960e0245b470000e4870f3dcd23de5e54: Status 404 returned error can't find the container with id d8c7d9cb616f4bb56429a1dd9bd232b960e0245b470000e4870f3dcd23de5e54
Jul 17 21:43:59.697670 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:59.697635    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp" event={"ID":"83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588","Type":"ContainerStarted","Data":"97111a096e372693cfe647a77d47fe771a9227d808b6853fc5cf4efc6e6a778e"}
Jul 17 21:43:59.697843 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:59.697686    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp" event={"ID":"83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588","Type":"ContainerStarted","Data":"d8c7d9cb616f4bb56429a1dd9bd232b960e0245b470000e4870f3dcd23de5e54"}
Jul 17 21:43:59.714070 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:43:59.714011    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp" podStartSLOduration=1.713988809 podStartE2EDuration="1.713988809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:43:59.71271909 +0000 UTC m=+2153.472285782" watchObservedRunningTime="2026-07-17 21:43:59.713988809 +0000 UTC m=+2153.473555520"
Jul 17 21:44:00.602263 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:00.602214    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:44:00.704969 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:00.704911    2569 generic.go:356] "Generic (PLEG): container finished" podID="83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588" containerID="97111a096e372693cfe647a77d47fe771a9227d808b6853fc5cf4efc6e6a778e" exitCode=6
Jul 17 21:44:00.705177 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:00.705010    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp" event={"ID":"83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588","Type":"ContainerDied","Data":"97111a096e372693cfe647a77d47fe771a9227d808b6853fc5cf4efc6e6a778e"}
Jul 17 21:44:00.705177 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:00.705156    2569 scope.go:117] "RemoveContainer" containerID="97111a096e372693cfe647a77d47fe771a9227d808b6853fc5cf4efc6e6a778e"
Jul 17 21:44:00.707162 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:00.707134    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:44:01.712414 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:01.712361    2569 generic.go:356] "Generic (PLEG): container finished" podID="83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588" containerID="f8845870402ffefbbf52bd62d1921076ac8a85ef956e99e431c13c0199401e45" exitCode=6
Jul 17 21:44:01.712913 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:01.712435    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp" event={"ID":"83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588","Type":"ContainerDied","Data":"f8845870402ffefbbf52bd62d1921076ac8a85ef956e99e431c13c0199401e45"}
Jul 17 21:44:01.712913 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:01.712493    2569 scope.go:117] "RemoveContainer" containerID="97111a096e372693cfe647a77d47fe771a9227d808b6853fc5cf4efc6e6a778e"
Jul 17 21:44:01.712913 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:01.712626    2569 scope.go:117] "RemoveContainer" containerID="f8845870402ffefbbf52bd62d1921076ac8a85ef956e99e431c13c0199401e45"
Jul 17 21:44:01.712913 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:44:01.712853    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp_odh-ai-gateway-infra(83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp" podUID="83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588"
Jul 17 21:44:02.719748 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:02.719719    2569 scope.go:117] "RemoveContainer" containerID="f8845870402ffefbbf52bd62d1921076ac8a85ef956e99e431c13c0199401e45"
Jul 17 21:44:02.720131 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:44:02.719924    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp_odh-ai-gateway-infra(83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp" podUID="83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588"
Jul 17 21:44:09.739489 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:09.739458    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:44:09.840208 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:09.840173    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:44:17.651157 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:17.651126    2569 scope.go:117] "RemoveContainer" containerID="f8845870402ffefbbf52bd62d1921076ac8a85ef956e99e431c13c0199401e45"
Jul 17 21:44:18.810927 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:18.810891    2569 generic.go:356] "Generic (PLEG): container finished" podID="83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588" containerID="d02039596b148a2055e27daf7ae6275dc2dcaf57102f485a4737f787ee565547" exitCode=6
Jul 17 21:44:18.811366 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:18.810962    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp" event={"ID":"83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588","Type":"ContainerDied","Data":"d02039596b148a2055e27daf7ae6275dc2dcaf57102f485a4737f787ee565547"}
Jul 17 21:44:18.811366 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:18.811019    2569 scope.go:117] "RemoveContainer" containerID="f8845870402ffefbbf52bd62d1921076ac8a85ef956e99e431c13c0199401e45"
Jul 17 21:44:18.811366 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:18.811339    2569 scope.go:117] "RemoveContainer" containerID="d02039596b148a2055e27daf7ae6275dc2dcaf57102f485a4737f787ee565547"
Jul 17 21:44:18.811589 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:44:18.811567    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp_odh-ai-gateway-infra(83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp" podUID="83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588"
Jul 17 21:44:19.842085 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:19.842049    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp"]
Jul 17 21:44:19.982973 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:19.982942    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp"
Jul 17 21:44:20.069181 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:20.069146    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588-maas-api-service-ca\") pod \"83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588\" (UID: \"83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588\") "
Jul 17 21:44:20.069552 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:20.069528    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588" (UID: "83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:44:20.170857 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:20.170769    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:44:20.825701 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:20.825663    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp" event={"ID":"83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588","Type":"ContainerDied","Data":"d8c7d9cb616f4bb56429a1dd9bd232b960e0245b470000e4870f3dcd23de5e54"}
Jul 17 21:44:20.825701 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:20.825686    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp"
Jul 17 21:44:20.825937 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:20.825726    2569 scope.go:117] "RemoveContainer" containerID="d02039596b148a2055e27daf7ae6275dc2dcaf57102f485a4737f787ee565547"
Jul 17 21:44:20.840062 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:20.840035    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp"]
Jul 17 21:44:20.846181 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:20.846158    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5cwtp"]
Jul 17 21:44:22.657634 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:22.657600    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588" path="/var/lib/kubelet/pods/83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588/volumes"
Jul 17 21:44:27.077646 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.077602    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j"]
Jul 17 21:44:27.078176 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.078156    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588" containerName="revoke-keys"
Jul 17 21:44:27.078262 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.078179    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588" containerName="revoke-keys"
Jul 17 21:44:27.078262 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.078193    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588" containerName="revoke-keys"
Jul 17 21:44:27.078262 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.078202    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588" containerName="revoke-keys"
Jul 17 21:44:27.078262 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.078221    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c3bbfd-7b8e-4b8f-848c-e38acf68a38c" containerName="revoke-keys"
Jul 17 21:44:27.078262 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.078230    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c3bbfd-7b8e-4b8f-848c-e38acf68a38c" containerName="revoke-keys"
Jul 17 21:44:27.078262 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.078243    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588" containerName="revoke-keys"
Jul 17 21:44:27.078262 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.078252    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588" containerName="revoke-keys"
Jul 17 21:44:27.078627 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.078383    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588" containerName="revoke-keys"
Jul 17 21:44:27.078627 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.078395    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c3bbfd-7b8e-4b8f-848c-e38acf68a38c" containerName="revoke-keys"
Jul 17 21:44:27.078627 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.078420    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588" containerName="revoke-keys"
Jul 17 21:44:27.082762 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.082737    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j"
Jul 17 21:44:27.084593 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.084573    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:44:27.084716 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.084634    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:44:27.088914 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.088887    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j"]
Jul 17 21:44:27.237482 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.237437    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/603c87cf-a454-40d2-a271-b772c590499f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j\" (UID: \"603c87cf-a454-40d2-a271-b772c590499f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j"
Jul 17 21:44:27.338542 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.338514    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/603c87cf-a454-40d2-a271-b772c590499f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j\" (UID: \"603c87cf-a454-40d2-a271-b772c590499f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j"
Jul 17 21:44:27.339136 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.339115    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/603c87cf-a454-40d2-a271-b772c590499f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j\" (UID: \"603c87cf-a454-40d2-a271-b772c590499f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j"
Jul 17 21:44:27.394565 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.394535    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j"
Jul 17 21:44:27.531158 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.531126    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j"]
Jul 17 21:44:27.532863 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:44:27.532837    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603c87cf_a454_40d2_a271_b772c590499f.slice/crio-7c8d5c3daa2cf61882e4f5330697334f7a83ea6c7cb561d5b7f758578c5a069e WatchSource:0}: Error finding container 7c8d5c3daa2cf61882e4f5330697334f7a83ea6c7cb561d5b7f758578c5a069e: Status 404 returned error can't find the container with id 7c8d5c3daa2cf61882e4f5330697334f7a83ea6c7cb561d5b7f758578c5a069e
Jul 17 21:44:27.877331 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.877299    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j" event={"ID":"603c87cf-a454-40d2-a271-b772c590499f","Type":"ContainerStarted","Data":"b7bb1b366bb81162fef20169652ae8c3eb428e9d70d11ec6e306c85d485e9db3"}
Jul 17 21:44:27.877485 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.877354    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j" event={"ID":"603c87cf-a454-40d2-a271-b772c590499f","Type":"ContainerStarted","Data":"7c8d5c3daa2cf61882e4f5330697334f7a83ea6c7cb561d5b7f758578c5a069e"}
Jul 17 21:44:27.896109 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:27.896052    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j" podStartSLOduration=0.896035458 podStartE2EDuration="896.035458ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:44:27.893619365 +0000 UTC m=+2181.653186058" watchObservedRunningTime="2026-07-17 21:44:27.896035458 +0000 UTC m=+2181.655602156"
Jul 17 21:44:28.884469 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:28.884434    2569 generic.go:356] "Generic (PLEG): container finished" podID="603c87cf-a454-40d2-a271-b772c590499f" containerID="b7bb1b366bb81162fef20169652ae8c3eb428e9d70d11ec6e306c85d485e9db3" exitCode=7
Jul 17 21:44:28.884875 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:28.884528    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j" event={"ID":"603c87cf-a454-40d2-a271-b772c590499f","Type":"ContainerDied","Data":"b7bb1b366bb81162fef20169652ae8c3eb428e9d70d11ec6e306c85d485e9db3"}
Jul 17 21:44:28.884875 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:28.884734    2569 scope.go:117] "RemoveContainer" containerID="b7bb1b366bb81162fef20169652ae8c3eb428e9d70d11ec6e306c85d485e9db3"
Jul 17 21:44:29.892736 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:29.892704    2569 generic.go:356] "Generic (PLEG): container finished" podID="603c87cf-a454-40d2-a271-b772c590499f" containerID="18fb2dcd1740e5852f1a15f30cea1ffbc48020bde7c8258f94f3e24471124c91" exitCode=7
Jul 17 21:44:29.893179 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:29.892787    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j" event={"ID":"603c87cf-a454-40d2-a271-b772c590499f","Type":"ContainerDied","Data":"18fb2dcd1740e5852f1a15f30cea1ffbc48020bde7c8258f94f3e24471124c91"}
Jul 17 21:44:29.893179 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:29.892850    2569 scope.go:117] "RemoveContainer" containerID="b7bb1b366bb81162fef20169652ae8c3eb428e9d70d11ec6e306c85d485e9db3"
Jul 17 21:44:29.893179 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:29.893102    2569 scope.go:117] "RemoveContainer" containerID="18fb2dcd1740e5852f1a15f30cea1ffbc48020bde7c8258f94f3e24471124c91"
Jul 17 21:44:29.893318 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:44:29.893298    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j_odh-ai-gateway-infra(603c87cf-a454-40d2-a271-b772c590499f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j" podUID="603c87cf-a454-40d2-a271-b772c590499f"
Jul 17 21:44:30.133932 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:30.133897    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z"]
Jul 17 21:44:30.134364 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:30.134352    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="83cc5d8a-a0f7-4c7a-9e3b-412c9e36a588" containerName="revoke-keys"
Jul 17 21:44:30.138552 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:30.138528    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z"
Jul 17 21:44:30.153080 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:30.153050    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z"]
Jul 17 21:44:30.163870 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:30.163838    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f408b1-0faa-4e2c-bc8e-f5528e2c373f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-th86z\" (UID: \"10f408b1-0faa-4e2c-bc8e-f5528e2c373f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z"
Jul 17 21:44:30.264483 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:30.264443    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f408b1-0faa-4e2c-bc8e-f5528e2c373f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-th86z\" (UID: \"10f408b1-0faa-4e2c-bc8e-f5528e2c373f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z"
Jul 17 21:44:30.265072 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:30.265049    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f408b1-0faa-4e2c-bc8e-f5528e2c373f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-th86z\" (UID: \"10f408b1-0faa-4e2c-bc8e-f5528e2c373f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z"
Jul 17 21:44:30.453632 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:30.453537    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z"
Jul 17 21:44:30.589935 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:30.589907    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z"]
Jul 17 21:44:30.592184 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:44:30.592155    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f408b1_0faa_4e2c_bc8e_f5528e2c373f.slice/crio-532d4225beddc78599be259046585cc07a5223a5603edd2f39944dae89ab070c WatchSource:0}: Error finding container 532d4225beddc78599be259046585cc07a5223a5603edd2f39944dae89ab070c: Status 404 returned error can't find the container with id 532d4225beddc78599be259046585cc07a5223a5603edd2f39944dae89ab070c
Jul 17 21:44:30.899930 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:30.899898    2569 scope.go:117] "RemoveContainer" containerID="18fb2dcd1740e5852f1a15f30cea1ffbc48020bde7c8258f94f3e24471124c91"
Jul 17 21:44:30.900426 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:44:30.900124    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j_odh-ai-gateway-infra(603c87cf-a454-40d2-a271-b772c590499f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j" podUID="603c87cf-a454-40d2-a271-b772c590499f"
Jul 17 21:44:30.901337 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:30.901309    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z" event={"ID":"10f408b1-0faa-4e2c-bc8e-f5528e2c373f","Type":"ContainerStarted","Data":"640445eaf989c48ef12a3ff067d42d52a7a71ce5ef5482be4cd5ff0edd6b2ba1"}
Jul 17 21:44:30.901436 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:30.901362    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z" event={"ID":"10f408b1-0faa-4e2c-bc8e-f5528e2c373f","Type":"ContainerStarted","Data":"532d4225beddc78599be259046585cc07a5223a5603edd2f39944dae89ab070c"}
Jul 17 21:44:30.928842 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:30.928783    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z" podStartSLOduration=0.928767255 podStartE2EDuration="928.767255ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:44:30.926003715 +0000 UTC m=+2184.685570408" watchObservedRunningTime="2026-07-17 21:44:30.928767255 +0000 UTC m=+2184.688333948"
Jul 17 21:44:31.908560 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:31.908523    2569 generic.go:356] "Generic (PLEG): container finished" podID="10f408b1-0faa-4e2c-bc8e-f5528e2c373f" containerID="640445eaf989c48ef12a3ff067d42d52a7a71ce5ef5482be4cd5ff0edd6b2ba1" exitCode=6
Jul 17 21:44:31.908971 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:31.908615    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z" event={"ID":"10f408b1-0faa-4e2c-bc8e-f5528e2c373f","Type":"ContainerDied","Data":"640445eaf989c48ef12a3ff067d42d52a7a71ce5ef5482be4cd5ff0edd6b2ba1"}
Jul 17 21:44:31.908971 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:31.908771    2569 scope.go:117] "RemoveContainer" containerID="640445eaf989c48ef12a3ff067d42d52a7a71ce5ef5482be4cd5ff0edd6b2ba1"
Jul 17 21:44:32.236234 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:32.236193    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:44:32.915935 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:32.915898    2569 generic.go:356] "Generic (PLEG): container finished" podID="10f408b1-0faa-4e2c-bc8e-f5528e2c373f" containerID="1b5e7c14cba7234dd74966397400ecdd373472ceb02476ad2ee42c3249e8e6a0" exitCode=6
Jul 17 21:44:32.916432 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:32.915966    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z" event={"ID":"10f408b1-0faa-4e2c-bc8e-f5528e2c373f","Type":"ContainerDied","Data":"1b5e7c14cba7234dd74966397400ecdd373472ceb02476ad2ee42c3249e8e6a0"}
Jul 17 21:44:32.916432 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:32.916013    2569 scope.go:117] "RemoveContainer" containerID="640445eaf989c48ef12a3ff067d42d52a7a71ce5ef5482be4cd5ff0edd6b2ba1"
Jul 17 21:44:32.916432 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:32.916237    2569 scope.go:117] "RemoveContainer" containerID="1b5e7c14cba7234dd74966397400ecdd373472ceb02476ad2ee42c3249e8e6a0"
Jul 17 21:44:32.916615 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:44:32.916528    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-th86z_odh-ai-gateway-infra(10f408b1-0faa-4e2c-bc8e-f5528e2c373f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z" podUID="10f408b1-0faa-4e2c-bc8e-f5528e2c373f"
Jul 17 21:44:33.923218 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:33.923191    2569 scope.go:117] "RemoveContainer" containerID="1b5e7c14cba7234dd74966397400ecdd373472ceb02476ad2ee42c3249e8e6a0"
Jul 17 21:44:33.923683 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:44:33.923454    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-th86z_odh-ai-gateway-infra(10f408b1-0faa-4e2c-bc8e-f5528e2c373f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z" podUID="10f408b1-0faa-4e2c-bc8e-f5528e2c373f"
Jul 17 21:44:43.651235 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:43.651199    2569 scope.go:117] "RemoveContainer" containerID="18fb2dcd1740e5852f1a15f30cea1ffbc48020bde7c8258f94f3e24471124c91"
Jul 17 21:44:43.983017 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:43.982985    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j" event={"ID":"603c87cf-a454-40d2-a271-b772c590499f","Type":"ContainerStarted","Data":"78766fd6a506f7d6a55acb46a9beb6bd0e4f410f975043bf40b501afc2e32b42"}
Jul 17 21:44:44.990630 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:44.990597    2569 generic.go:356] "Generic (PLEG): container finished" podID="603c87cf-a454-40d2-a271-b772c590499f" containerID="78766fd6a506f7d6a55acb46a9beb6bd0e4f410f975043bf40b501afc2e32b42" exitCode=7
Jul 17 21:44:44.991211 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:44.990669    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j" event={"ID":"603c87cf-a454-40d2-a271-b772c590499f","Type":"ContainerDied","Data":"78766fd6a506f7d6a55acb46a9beb6bd0e4f410f975043bf40b501afc2e32b42"}
Jul 17 21:44:44.991211 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:44.990730    2569 scope.go:117] "RemoveContainer" containerID="18fb2dcd1740e5852f1a15f30cea1ffbc48020bde7c8258f94f3e24471124c91"
Jul 17 21:44:44.991211 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:44.991013    2569 scope.go:117] "RemoveContainer" containerID="78766fd6a506f7d6a55acb46a9beb6bd0e4f410f975043bf40b501afc2e32b42"
Jul 17 21:44:44.991387 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:44:44.991254    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j_odh-ai-gateway-infra(603c87cf-a454-40d2-a271-b772c590499f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j" podUID="603c87cf-a454-40d2-a271-b772c590499f"
Jul 17 21:44:45.014245 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:45.014208    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j"]
Jul 17 21:44:46.138057 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:46.138030    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j"
Jul 17 21:44:46.201125 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:46.201088    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/603c87cf-a454-40d2-a271-b772c590499f-maas-api-service-ca\") pod \"603c87cf-a454-40d2-a271-b772c590499f\" (UID: \"603c87cf-a454-40d2-a271-b772c590499f\") "
Jul 17 21:44:46.201456 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:46.201436    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603c87cf-a454-40d2-a271-b772c590499f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "603c87cf-a454-40d2-a271-b772c590499f" (UID: "603c87cf-a454-40d2-a271-b772c590499f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:44:46.302263 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:46.302178    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/603c87cf-a454-40d2-a271-b772c590499f-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:44:47.004034 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:47.003942    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j"
Jul 17 21:44:47.004034 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:47.003964    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j" event={"ID":"603c87cf-a454-40d2-a271-b772c590499f","Type":"ContainerDied","Data":"7c8d5c3daa2cf61882e4f5330697334f7a83ea6c7cb561d5b7f758578c5a069e"}
Jul 17 21:44:47.004267 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:47.004035    2569 scope.go:117] "RemoveContainer" containerID="78766fd6a506f7d6a55acb46a9beb6bd0e4f410f975043bf40b501afc2e32b42"
Jul 17 21:44:47.021608 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:47.021584    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j"]
Jul 17 21:44:47.031228 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:47.031204    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gnk9j"]
Jul 17 21:44:48.650973 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:48.650931    2569 scope.go:117] "RemoveContainer" containerID="1b5e7c14cba7234dd74966397400ecdd373472ceb02476ad2ee42c3249e8e6a0"
Jul 17 21:44:48.658152 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:48.658122    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="603c87cf-a454-40d2-a271-b772c590499f" path="/var/lib/kubelet/pods/603c87cf-a454-40d2-a271-b772c590499f/volumes"
Jul 17 21:44:49.022125 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:49.019493    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z" event={"ID":"10f408b1-0faa-4e2c-bc8e-f5528e2c373f","Type":"ContainerStarted","Data":"45fa7a8e057ee0fce33481668d8f16d8247473e7c9d76f1c64195a627cdbc8e5"}
Jul 17 21:44:50.027274 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:50.027239    2569 generic.go:356] "Generic (PLEG): container finished" podID="10f408b1-0faa-4e2c-bc8e-f5528e2c373f" containerID="45fa7a8e057ee0fce33481668d8f16d8247473e7c9d76f1c64195a627cdbc8e5" exitCode=6
Jul 17 21:44:50.027714 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:50.027302    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z" event={"ID":"10f408b1-0faa-4e2c-bc8e-f5528e2c373f","Type":"ContainerDied","Data":"45fa7a8e057ee0fce33481668d8f16d8247473e7c9d76f1c64195a627cdbc8e5"}
Jul 17 21:44:50.027714 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:50.027356    2569 scope.go:117] "RemoveContainer" containerID="1b5e7c14cba7234dd74966397400ecdd373472ceb02476ad2ee42c3249e8e6a0"
Jul 17 21:44:50.027714 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:50.027664    2569 scope.go:117] "RemoveContainer" containerID="45fa7a8e057ee0fce33481668d8f16d8247473e7c9d76f1c64195a627cdbc8e5"
Jul 17 21:44:50.027894 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:44:50.027872    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-th86z_odh-ai-gateway-infra(10f408b1-0faa-4e2c-bc8e-f5528e2c373f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z" podUID="10f408b1-0faa-4e2c-bc8e-f5528e2c373f"
Jul 17 21:44:50.050880 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:50.050852    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z"]
Jul 17 21:44:51.176920 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:51.176894    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z"
Jul 17 21:44:51.250721 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:51.250684    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f408b1-0faa-4e2c-bc8e-f5528e2c373f-maas-api-service-ca\") pod \"10f408b1-0faa-4e2c-bc8e-f5528e2c373f\" (UID: \"10f408b1-0faa-4e2c-bc8e-f5528e2c373f\") "
Jul 17 21:44:51.251063 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:51.251040    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f408b1-0faa-4e2c-bc8e-f5528e2c373f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "10f408b1-0faa-4e2c-bc8e-f5528e2c373f" (UID: "10f408b1-0faa-4e2c-bc8e-f5528e2c373f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:44:51.351549 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:51.351514    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f408b1-0faa-4e2c-bc8e-f5528e2c373f-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:44:52.041611 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:52.041576    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z"
Jul 17 21:44:52.041822 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:52.041621    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z" event={"ID":"10f408b1-0faa-4e2c-bc8e-f5528e2c373f","Type":"ContainerDied","Data":"532d4225beddc78599be259046585cc07a5223a5603edd2f39944dae89ab070c"}
Jul 17 21:44:52.041822 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:52.041720    2569 scope.go:117] "RemoveContainer" containerID="45fa7a8e057ee0fce33481668d8f16d8247473e7c9d76f1c64195a627cdbc8e5"
Jul 17 21:44:52.068936 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:52.068864    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z"]
Jul 17 21:44:52.072200 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:52.072174    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-th86z"]
Jul 17 21:44:52.657069 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:52.657033    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10f408b1-0faa-4e2c-bc8e-f5528e2c373f" path="/var/lib/kubelet/pods/10f408b1-0faa-4e2c-bc8e-f5528e2c373f/volumes"
Jul 17 21:44:58.978752 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:58.978704    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6"]
Jul 17 21:44:58.979215 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:58.979168    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10f408b1-0faa-4e2c-bc8e-f5528e2c373f" containerName="revoke-keys"
Jul 17 21:44:58.979215 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:58.979179    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="10f408b1-0faa-4e2c-bc8e-f5528e2c373f" containerName="revoke-keys"
Jul 17 21:44:58.979215 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:58.979192    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603c87cf-a454-40d2-a271-b772c590499f" containerName="revoke-keys"
Jul 17 21:44:58.979215 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:58.979199    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="603c87cf-a454-40d2-a271-b772c590499f" containerName="revoke-keys"
Jul 17 21:44:58.979442 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:58.979225    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603c87cf-a454-40d2-a271-b772c590499f" containerName="revoke-keys"
Jul 17 21:44:58.979442 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:58.979231    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="603c87cf-a454-40d2-a271-b772c590499f" containerName="revoke-keys"
Jul 17 21:44:58.979442 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:58.979241    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10f408b1-0faa-4e2c-bc8e-f5528e2c373f" containerName="revoke-keys"
Jul 17 21:44:58.979442 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:58.979247    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="10f408b1-0faa-4e2c-bc8e-f5528e2c373f" containerName="revoke-keys"
Jul 17 21:44:58.979442 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:58.979315    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="603c87cf-a454-40d2-a271-b772c590499f" containerName="revoke-keys"
Jul 17 21:44:58.979442 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:58.979323    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="10f408b1-0faa-4e2c-bc8e-f5528e2c373f" containerName="revoke-keys"
Jul 17 21:44:58.979442 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:58.979328    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="10f408b1-0faa-4e2c-bc8e-f5528e2c373f" containerName="revoke-keys"
Jul 17 21:44:58.979442 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:58.979334    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="603c87cf-a454-40d2-a271-b772c590499f" containerName="revoke-keys"
Jul 17 21:44:58.979442 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:58.979339    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="603c87cf-a454-40d2-a271-b772c590499f" containerName="revoke-keys"
Jul 17 21:44:58.983666 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:58.983640    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6"
Jul 17 21:44:58.985771 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:58.985747    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:44:58.985771 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:58.985763    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:44:58.989788 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:58.989764    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6"]
Jul 17 21:44:59.122613 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:59.122573    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cc0744-665e-4c37-a75c-cb96951cc2f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6\" (UID: \"a9cc0744-665e-4c37-a75c-cb96951cc2f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6"
Jul 17 21:44:59.223949 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:59.223909    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cc0744-665e-4c37-a75c-cb96951cc2f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6\" (UID: \"a9cc0744-665e-4c37-a75c-cb96951cc2f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6"
Jul 17 21:44:59.224633 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:59.224612    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cc0744-665e-4c37-a75c-cb96951cc2f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6\" (UID: \"a9cc0744-665e-4c37-a75c-cb96951cc2f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6"
Jul 17 21:44:59.296110 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:59.296016    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6"
Jul 17 21:44:59.435208 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:44:59.435182    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6"]
Jul 17 21:44:59.436772 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:44:59.436745    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cc0744_665e_4c37_a75c_cb96951cc2f0.slice/crio-617713e7838a3b00750d26295308b22789087a578231e665e8dfc18d2942c6c6 WatchSource:0}: Error finding container 617713e7838a3b00750d26295308b22789087a578231e665e8dfc18d2942c6c6: Status 404 returned error can't find the container with id 617713e7838a3b00750d26295308b22789087a578231e665e8dfc18d2942c6c6
Jul 17 21:45:00.089913 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:00.089873    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6" event={"ID":"a9cc0744-665e-4c37-a75c-cb96951cc2f0","Type":"ContainerStarted","Data":"09b87f5e85a4e01338fb149b3756323335057ad719b46c62b353d2321ac9c6c7"}
Jul 17 21:45:00.090279 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:00.089966    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6" event={"ID":"a9cc0744-665e-4c37-a75c-cb96951cc2f0","Type":"ContainerStarted","Data":"617713e7838a3b00750d26295308b22789087a578231e665e8dfc18d2942c6c6"}
Jul 17 21:45:00.106983 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:00.106927    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6" podStartSLOduration=2.106910611 podStartE2EDuration="2.106910611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:45:00.104547453 +0000 UTC m=+2213.864114146" watchObservedRunningTime="2026-07-17 21:45:00.106910611 +0000 UTC m=+2213.866477304"
Jul 17 21:45:00.138743 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:00.138709    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738745-qfv5x"]
Jul 17 21:45:00.139158 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:00.139145    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10f408b1-0faa-4e2c-bc8e-f5528e2c373f" containerName="revoke-keys"
Jul 17 21:45:00.139251 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:00.139160    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="10f408b1-0faa-4e2c-bc8e-f5528e2c373f" containerName="revoke-keys"
Jul 17 21:45:00.139251 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:00.139171    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603c87cf-a454-40d2-a271-b772c590499f" containerName="revoke-keys"
Jul 17 21:45:00.139251 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:00.139177    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="603c87cf-a454-40d2-a271-b772c590499f" containerName="revoke-keys"
Jul 17 21:45:00.139251 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:00.139236    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="10f408b1-0faa-4e2c-bc8e-f5528e2c373f" containerName="revoke-keys"
Jul 17 21:45:00.142679 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:00.142658    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs"]
Jul 17 21:45:00.142796 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:00.142775    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-qfv5x"
Jul 17 21:45:00.145959 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:00.145937    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs"
Jul 17 21:45:00.152741 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:00.152717    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738745-qfv5x"]
Jul 17 21:45:00.156148 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:00.155934    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-qfv5x"
Jul 17 21:45:00.156148 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:00.156014    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs"]
Jul 17 21:45:00.165749 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:00.165724    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs"
Jul 17 21:45:00.333182 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:00.333157    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs"]
Jul 17 21:45:00.334849 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:45:00.334820    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65150b42_1037_4ec5_92fc_e332d7455706.slice/crio-160be6146420ca03a67395ceeb3282c03efcd9fad632f65a4d7cd8a992e1a0f4 WatchSource:0}: Error finding container 160be6146420ca03a67395ceeb3282c03efcd9fad632f65a4d7cd8a992e1a0f4: Status 404 returned error can't find the container with id 160be6146420ca03a67395ceeb3282c03efcd9fad632f65a4d7cd8a992e1a0f4
Jul 17 21:45:00.515486 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:00.515462    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738745-qfv5x"]
Jul 17 21:45:00.517635 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:45:00.517606    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b3e52b_5945_4c17_8bb7_92aa7bb119e6.slice/crio-c76052142515ff4f220dae5d079f047d22137ac11276700ecb369da8ea646078 WatchSource:0}: Error finding container c76052142515ff4f220dae5d079f047d22137ac11276700ecb369da8ea646078: Status 404 returned error can't find the container with id c76052142515ff4f220dae5d079f047d22137ac11276700ecb369da8ea646078
Jul 17 21:45:01.096668 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:01.096632    2569 generic.go:356] "Generic (PLEG): container finished" podID="a9cc0744-665e-4c37-a75c-cb96951cc2f0" containerID="09b87f5e85a4e01338fb149b3756323335057ad719b46c62b353d2321ac9c6c7" exitCode=6
Jul 17 21:45:01.097465 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:01.096715    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6" event={"ID":"a9cc0744-665e-4c37-a75c-cb96951cc2f0","Type":"ContainerDied","Data":"09b87f5e85a4e01338fb149b3756323335057ad719b46c62b353d2321ac9c6c7"}
Jul 17 21:45:01.097465 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:01.096914    2569 scope.go:117] "RemoveContainer" containerID="09b87f5e85a4e01338fb149b3756323335057ad719b46c62b353d2321ac9c6c7"
Jul 17 21:45:01.098441 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:01.098419    2569 generic.go:356] "Generic (PLEG): container finished" podID="65150b42-1037-4ec5-92fc-e332d7455706" containerID="f8fdb5c86764c9ee6f560dcb42f6f478332425026b30016abf0b4c5069b64040" exitCode=7
Jul 17 21:45:01.098544 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:01.098509    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs" event={"ID":"65150b42-1037-4ec5-92fc-e332d7455706","Type":"ContainerDied","Data":"f8fdb5c86764c9ee6f560dcb42f6f478332425026b30016abf0b4c5069b64040"}
Jul 17 21:45:01.098641 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:01.098563    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs" event={"ID":"65150b42-1037-4ec5-92fc-e332d7455706","Type":"ContainerStarted","Data":"160be6146420ca03a67395ceeb3282c03efcd9fad632f65a4d7cd8a992e1a0f4"}
Jul 17 21:45:01.098641 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:01.098610    2569 scope.go:117] "RemoveContainer" containerID="f8fdb5c86764c9ee6f560dcb42f6f478332425026b30016abf0b4c5069b64040"
Jul 17 21:45:01.100110 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:01.100090    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-qfv5x" event={"ID":"a6b3e52b-5945-4c17-8bb7-92aa7bb119e6","Type":"ContainerStarted","Data":"52eead77834a3705521ad943a6905caf6d72468c813d5e0a9e2d1fc6f612dfe8"}
Jul 17 21:45:01.100206 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:01.100129    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-qfv5x" event={"ID":"a6b3e52b-5945-4c17-8bb7-92aa7bb119e6","Type":"ContainerStarted","Data":"c76052142515ff4f220dae5d079f047d22137ac11276700ecb369da8ea646078"}
Jul 17 21:45:01.139246 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:01.139197    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-qfv5x" podStartSLOduration=1.139182432 podStartE2EDuration="1.139182432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:45:01.137321844 +0000 UTC m=+2214.896888538" watchObservedRunningTime="2026-07-17 21:45:01.139182432 +0000 UTC m=+2214.898749126"
Jul 17 21:45:02.107968 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:02.107932    2569 generic.go:356] "Generic (PLEG): container finished" podID="a9cc0744-665e-4c37-a75c-cb96951cc2f0" containerID="298c3f18e0bed3c12292473f47ea1c78ecd80c1894c322b585514455b0b191a9" exitCode=6
Jul 17 21:45:02.108446 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:02.108011    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6" event={"ID":"a9cc0744-665e-4c37-a75c-cb96951cc2f0","Type":"ContainerDied","Data":"298c3f18e0bed3c12292473f47ea1c78ecd80c1894c322b585514455b0b191a9"}
Jul 17 21:45:02.108446 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:02.108065    2569 scope.go:117] "RemoveContainer" containerID="09b87f5e85a4e01338fb149b3756323335057ad719b46c62b353d2321ac9c6c7"
Jul 17 21:45:02.108446 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:02.108262    2569 scope.go:117] "RemoveContainer" containerID="298c3f18e0bed3c12292473f47ea1c78ecd80c1894c322b585514455b0b191a9"
Jul 17 21:45:02.108571 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:45:02.108456    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6_odh-ai-gateway-infra(a9cc0744-665e-4c37-a75c-cb96951cc2f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6" podUID="a9cc0744-665e-4c37-a75c-cb96951cc2f0"
Jul 17 21:45:02.109934 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:02.109908    2569 generic.go:356] "Generic (PLEG): container finished" podID="65150b42-1037-4ec5-92fc-e332d7455706" containerID="8779cc3bc0da32f4627e0d8e6d85dc241c26015e0f3efab36b65937f39d57ff9" exitCode=7
Jul 17 21:45:02.110053 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:02.109981    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs" event={"ID":"65150b42-1037-4ec5-92fc-e332d7455706","Type":"ContainerDied","Data":"8779cc3bc0da32f4627e0d8e6d85dc241c26015e0f3efab36b65937f39d57ff9"}
Jul 17 21:45:02.110133 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:02.110118    2569 scope.go:117] "RemoveContainer" containerID="8779cc3bc0da32f4627e0d8e6d85dc241c26015e0f3efab36b65937f39d57ff9"
Jul 17 21:45:02.110342 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:45:02.110317    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs_odh-ai-gateway-infra(65150b42-1037-4ec5-92fc-e332d7455706)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs" podUID="65150b42-1037-4ec5-92fc-e332d7455706"
Jul 17 21:45:02.111382 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:02.111364    2569 generic.go:356] "Generic (PLEG): container finished" podID="a6b3e52b-5945-4c17-8bb7-92aa7bb119e6" containerID="52eead77834a3705521ad943a6905caf6d72468c813d5e0a9e2d1fc6f612dfe8" exitCode=0
Jul 17 21:45:02.111483 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:02.111434    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-qfv5x" event={"ID":"a6b3e52b-5945-4c17-8bb7-92aa7bb119e6","Type":"ContainerDied","Data":"52eead77834a3705521ad943a6905caf6d72468c813d5e0a9e2d1fc6f612dfe8"}
Jul 17 21:45:02.119631 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:02.119612    2569 scope.go:117] "RemoveContainer" containerID="f8fdb5c86764c9ee6f560dcb42f6f478332425026b30016abf0b4c5069b64040"
Jul 17 21:45:03.118282 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:03.118248    2569 scope.go:117] "RemoveContainer" containerID="8779cc3bc0da32f4627e0d8e6d85dc241c26015e0f3efab36b65937f39d57ff9"
Jul 17 21:45:03.118761 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:45:03.118516    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs_odh-ai-gateway-infra(65150b42-1037-4ec5-92fc-e332d7455706)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs" podUID="65150b42-1037-4ec5-92fc-e332d7455706"
Jul 17 21:45:03.119978 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:03.119957    2569 scope.go:117] "RemoveContainer" containerID="298c3f18e0bed3c12292473f47ea1c78ecd80c1894c322b585514455b0b191a9"
Jul 17 21:45:03.120339 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:45:03.120130    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6_odh-ai-gateway-infra(a9cc0744-665e-4c37-a75c-cb96951cc2f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6" podUID="a9cc0744-665e-4c37-a75c-cb96951cc2f0"
Jul 17 21:45:03.261618 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:03.261588    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-qfv5x"
Jul 17 21:45:04.128168 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:04.128134    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-qfv5x"
Jul 17 21:45:04.128585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:04.128169    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-qfv5x" event={"ID":"a6b3e52b-5945-4c17-8bb7-92aa7bb119e6","Type":"ContainerDied","Data":"c76052142515ff4f220dae5d079f047d22137ac11276700ecb369da8ea646078"}
Jul 17 21:45:04.128585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:04.128212    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c76052142515ff4f220dae5d079f047d22137ac11276700ecb369da8ea646078"
Jul 17 21:45:14.651037 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:14.650992    2569 scope.go:117] "RemoveContainer" containerID="8779cc3bc0da32f4627e0d8e6d85dc241c26015e0f3efab36b65937f39d57ff9"
Jul 17 21:45:14.651459 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:14.651123    2569 scope.go:117] "RemoveContainer" containerID="298c3f18e0bed3c12292473f47ea1c78ecd80c1894c322b585514455b0b191a9"
Jul 17 21:45:15.199527 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:15.199491    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs" event={"ID":"65150b42-1037-4ec5-92fc-e332d7455706","Type":"ContainerStarted","Data":"bed25daeeb58d1752c9f0a3bdb731cbab42530ea87dd833e5303c2d192e163d1"}
Jul 17 21:45:15.201396 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:15.201373    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6" event={"ID":"a9cc0744-665e-4c37-a75c-cb96951cc2f0","Type":"ContainerStarted","Data":"b210e60b4a71dea7bceeb68a88760ce23a1c7b7c9197674de5e14e3af93af54c"}
Jul 17 21:45:15.216235 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:15.216182    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs" podStartSLOduration=15.216166203 podStartE2EDuration="15.216166203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:45:15.213313574 +0000 UTC m=+2228.972880268" watchObservedRunningTime="2026-07-17 21:45:15.216166203 +0000 UTC m=+2228.975732894"
Jul 17 21:45:16.208558 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:16.208520    2569 generic.go:356] "Generic (PLEG): container finished" podID="65150b42-1037-4ec5-92fc-e332d7455706" containerID="bed25daeeb58d1752c9f0a3bdb731cbab42530ea87dd833e5303c2d192e163d1" exitCode=7
Jul 17 21:45:16.209143 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:16.208595    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs" event={"ID":"65150b42-1037-4ec5-92fc-e332d7455706","Type":"ContainerDied","Data":"bed25daeeb58d1752c9f0a3bdb731cbab42530ea87dd833e5303c2d192e163d1"}
Jul 17 21:45:16.209143 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:16.208667    2569 scope.go:117] "RemoveContainer" containerID="8779cc3bc0da32f4627e0d8e6d85dc241c26015e0f3efab36b65937f39d57ff9"
Jul 17 21:45:16.209143 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:16.208964    2569 scope.go:117] "RemoveContainer" containerID="bed25daeeb58d1752c9f0a3bdb731cbab42530ea87dd833e5303c2d192e163d1"
Jul 17 21:45:16.209315 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:45:16.209245    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs_odh-ai-gateway-infra(65150b42-1037-4ec5-92fc-e332d7455706)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs" podUID="65150b42-1037-4ec5-92fc-e332d7455706"
Jul 17 21:45:16.210664 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:16.210640    2569 generic.go:356] "Generic (PLEG): container finished" podID="a9cc0744-665e-4c37-a75c-cb96951cc2f0" containerID="b210e60b4a71dea7bceeb68a88760ce23a1c7b7c9197674de5e14e3af93af54c" exitCode=6
Jul 17 21:45:16.210767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:16.210692    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6" event={"ID":"a9cc0744-665e-4c37-a75c-cb96951cc2f0","Type":"ContainerDied","Data":"b210e60b4a71dea7bceeb68a88760ce23a1c7b7c9197674de5e14e3af93af54c"}
Jul 17 21:45:16.210930 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:16.210918    2569 scope.go:117] "RemoveContainer" containerID="b210e60b4a71dea7bceeb68a88760ce23a1c7b7c9197674de5e14e3af93af54c"
Jul 17 21:45:16.211100 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:45:16.211086    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6_odh-ai-gateway-infra(a9cc0744-665e-4c37-a75c-cb96951cc2f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6" podUID="a9cc0744-665e-4c37-a75c-cb96951cc2f0"
Jul 17 21:45:16.219793 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:16.219775    2569 scope.go:117] "RemoveContainer" containerID="298c3f18e0bed3c12292473f47ea1c78ecd80c1894c322b585514455b0b191a9"
Jul 17 21:45:16.224513 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:16.224491    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs"]
Jul 17 21:45:16.238993 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:16.238965    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6"]
Jul 17 21:45:17.377619 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:17.377595    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6"
Jul 17 21:45:17.379710 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:17.379687    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs"
Jul 17 21:45:17.385654 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:17.385622    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cc0744-665e-4c37-a75c-cb96951cc2f0-maas-api-service-ca\") pod \"a9cc0744-665e-4c37-a75c-cb96951cc2f0\" (UID: \"a9cc0744-665e-4c37-a75c-cb96951cc2f0\") "
Jul 17 21:45:17.385926 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:17.385904    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9cc0744-665e-4c37-a75c-cb96951cc2f0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a9cc0744-665e-4c37-a75c-cb96951cc2f0" (UID: "a9cc0744-665e-4c37-a75c-cb96951cc2f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:45:17.486501 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:17.486388    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cc0744-665e-4c37-a75c-cb96951cc2f0-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:45:18.226574 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:18.226541    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6"
Jul 17 21:45:18.226763 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:18.226574    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6" event={"ID":"a9cc0744-665e-4c37-a75c-cb96951cc2f0","Type":"ContainerDied","Data":"617713e7838a3b00750d26295308b22789087a578231e665e8dfc18d2942c6c6"}
Jul 17 21:45:18.226763 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:18.226637    2569 scope.go:117] "RemoveContainer" containerID="b210e60b4a71dea7bceeb68a88760ce23a1c7b7c9197674de5e14e3af93af54c"
Jul 17 21:45:18.228008 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:18.227918    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs"
Jul 17 21:45:18.228008 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:18.227925    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs" event={"ID":"65150b42-1037-4ec5-92fc-e332d7455706","Type":"ContainerDied","Data":"160be6146420ca03a67395ceeb3282c03efcd9fad632f65a4d7cd8a992e1a0f4"}
Jul 17 21:45:18.237235 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:18.237213    2569 scope.go:117] "RemoveContainer" containerID="bed25daeeb58d1752c9f0a3bdb731cbab42530ea87dd833e5303c2d192e163d1"
Jul 17 21:45:18.252232 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:18.252209    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6"]
Jul 17 21:45:18.257557 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:18.257531    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-r67m6"]
Jul 17 21:45:18.266997 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:18.266974    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs"]
Jul 17 21:45:18.272205 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:18.272182    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db4eb08b-29738745-f4zrs"]
Jul 17 21:45:18.657512 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:18.657471    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65150b42-1037-4ec5-92fc-e332d7455706" path="/var/lib/kubelet/pods/65150b42-1037-4ec5-92fc-e332d7455706/volumes"
Jul 17 21:45:18.658070 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:18.658050    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9cc0744-665e-4c37-a75c-cb96951cc2f0" path="/var/lib/kubelet/pods/a9cc0744-665e-4c37-a75c-cb96951cc2f0/volumes"
Jul 17 21:45:23.187776 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:45:23.187737    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:17.177189 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.177149    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt"]
Jul 17 21:46:17.177644 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.177614    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65150b42-1037-4ec5-92fc-e332d7455706" containerName="cleanup"
Jul 17 21:46:17.177644 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.177628    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="65150b42-1037-4ec5-92fc-e332d7455706" containerName="cleanup"
Jul 17 21:46:17.177644 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.177637    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9cc0744-665e-4c37-a75c-cb96951cc2f0" containerName="revoke-keys"
Jul 17 21:46:17.177644 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.177643    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9cc0744-665e-4c37-a75c-cb96951cc2f0" containerName="revoke-keys"
Jul 17 21:46:17.177803 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.177652    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6b3e52b-5945-4c17-8bb7-92aa7bb119e6" containerName="cleanup"
Jul 17 21:46:17.177803 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.177658    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6b3e52b-5945-4c17-8bb7-92aa7bb119e6" containerName="cleanup"
Jul 17 21:46:17.177803 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.177679    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9cc0744-665e-4c37-a75c-cb96951cc2f0" containerName="revoke-keys"
Jul 17 21:46:17.177803 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.177685    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9cc0744-665e-4c37-a75c-cb96951cc2f0" containerName="revoke-keys"
Jul 17 21:46:17.177803 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.177690    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65150b42-1037-4ec5-92fc-e332d7455706" containerName="cleanup"
Jul 17 21:46:17.177803 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.177695    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="65150b42-1037-4ec5-92fc-e332d7455706" containerName="cleanup"
Jul 17 21:46:17.177803 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.177763    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="65150b42-1037-4ec5-92fc-e332d7455706" containerName="cleanup"
Jul 17 21:46:17.177803 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.177771    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="65150b42-1037-4ec5-92fc-e332d7455706" containerName="cleanup"
Jul 17 21:46:17.177803 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.177777    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6b3e52b-5945-4c17-8bb7-92aa7bb119e6" containerName="cleanup"
Jul 17 21:46:17.177803 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.177784    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9cc0744-665e-4c37-a75c-cb96951cc2f0" containerName="revoke-keys"
Jul 17 21:46:17.177803 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.177791    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9cc0744-665e-4c37-a75c-cb96951cc2f0" containerName="revoke-keys"
Jul 17 21:46:17.177803 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.177796    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9cc0744-665e-4c37-a75c-cb96951cc2f0" containerName="revoke-keys"
Jul 17 21:46:17.182241 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.182223    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt"
Jul 17 21:46:17.184444 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.184420    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:46:17.184549 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.184475    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:46:17.188979 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.188952    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt"]
Jul 17 21:46:17.202786 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.202749    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e7549d-6c49-4fb9-a868-5c94d29f279a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt\" (UID: \"05e7549d-6c49-4fb9-a868-5c94d29f279a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt"
Jul 17 21:46:17.303285 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.303250    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e7549d-6c49-4fb9-a868-5c94d29f279a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt\" (UID: \"05e7549d-6c49-4fb9-a868-5c94d29f279a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt"
Jul 17 21:46:17.303998 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.303977    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e7549d-6c49-4fb9-a868-5c94d29f279a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt\" (UID: \"05e7549d-6c49-4fb9-a868-5c94d29f279a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt"
Jul 17 21:46:17.494657 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.494572    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt"
Jul 17 21:46:17.640214 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:17.640184    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt"]
Jul 17 21:46:17.642390 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:46:17.642349    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e7549d_6c49_4fb9_a868_5c94d29f279a.slice/crio-eb335dc36b83c85eadf697422fe27580f62a59ac9892b0f9e1a4e73b26db67fc WatchSource:0}: Error finding container eb335dc36b83c85eadf697422fe27580f62a59ac9892b0f9e1a4e73b26db67fc: Status 404 returned error can't find the container with id eb335dc36b83c85eadf697422fe27580f62a59ac9892b0f9e1a4e73b26db67fc
Jul 17 21:46:18.583985 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:18.583953    2569 generic.go:356] "Generic (PLEG): container finished" podID="05e7549d-6c49-4fb9-a868-5c94d29f279a" containerID="aa23ae9ae0f8ac61d42a6b55b177fc92b92583c50ab94e12d1908828a7b9f5f1" exitCode=7
Jul 17 21:46:18.584381 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:18.584042    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt" event={"ID":"05e7549d-6c49-4fb9-a868-5c94d29f279a","Type":"ContainerDied","Data":"aa23ae9ae0f8ac61d42a6b55b177fc92b92583c50ab94e12d1908828a7b9f5f1"}
Jul 17 21:46:18.584381 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:18.584094    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt" event={"ID":"05e7549d-6c49-4fb9-a868-5c94d29f279a","Type":"ContainerStarted","Data":"eb335dc36b83c85eadf697422fe27580f62a59ac9892b0f9e1a4e73b26db67fc"}
Jul 17 21:46:18.584381 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:18.584205    2569 scope.go:117] "RemoveContainer" containerID="aa23ae9ae0f8ac61d42a6b55b177fc92b92583c50ab94e12d1908828a7b9f5f1"
Jul 17 21:46:19.591713 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:19.591680    2569 generic.go:356] "Generic (PLEG): container finished" podID="05e7549d-6c49-4fb9-a868-5c94d29f279a" containerID="6c95f00dbc710e377a13cdf2574410659e41e0ad0a0543375c53599cceb1bc67" exitCode=7
Jul 17 21:46:19.592203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:19.591763    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt" event={"ID":"05e7549d-6c49-4fb9-a868-5c94d29f279a","Type":"ContainerDied","Data":"6c95f00dbc710e377a13cdf2574410659e41e0ad0a0543375c53599cceb1bc67"}
Jul 17 21:46:19.592203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:19.591822    2569 scope.go:117] "RemoveContainer" containerID="aa23ae9ae0f8ac61d42a6b55b177fc92b92583c50ab94e12d1908828a7b9f5f1"
Jul 17 21:46:19.592203 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:19.592037    2569 scope.go:117] "RemoveContainer" containerID="6c95f00dbc710e377a13cdf2574410659e41e0ad0a0543375c53599cceb1bc67"
Jul 17 21:46:19.592358 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:46:19.592290    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt_odh-ai-gateway-infra(05e7549d-6c49-4fb9-a868-5c94d29f279a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt" podUID="05e7549d-6c49-4fb9-a868-5c94d29f279a"
Jul 17 21:46:20.600520 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:20.600490    2569 scope.go:117] "RemoveContainer" containerID="6c95f00dbc710e377a13cdf2574410659e41e0ad0a0543375c53599cceb1bc67"
Jul 17 21:46:20.600873 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:46:20.600683    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt_odh-ai-gateway-infra(05e7549d-6c49-4fb9-a868-5c94d29f279a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt" podUID="05e7549d-6c49-4fb9-a868-5c94d29f279a"
Jul 17 21:46:35.651003 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:35.650965    2569 scope.go:117] "RemoveContainer" containerID="6c95f00dbc710e377a13cdf2574410659e41e0ad0a0543375c53599cceb1bc67"
Jul 17 21:46:36.693278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:36.693245    2569 generic.go:356] "Generic (PLEG): container finished" podID="05e7549d-6c49-4fb9-a868-5c94d29f279a" containerID="a6fb4b32c7c300718394b4085464e6c5e601128bff83ed047826367214c5f993" exitCode=7
Jul 17 21:46:36.693743 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:36.693318    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt" event={"ID":"05e7549d-6c49-4fb9-a868-5c94d29f279a","Type":"ContainerDied","Data":"a6fb4b32c7c300718394b4085464e6c5e601128bff83ed047826367214c5f993"}
Jul 17 21:46:36.693743 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:36.693380    2569 scope.go:117] "RemoveContainer" containerID="6c95f00dbc710e377a13cdf2574410659e41e0ad0a0543375c53599cceb1bc67"
Jul 17 21:46:36.693743 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:36.693670    2569 scope.go:117] "RemoveContainer" containerID="a6fb4b32c7c300718394b4085464e6c5e601128bff83ed047826367214c5f993"
Jul 17 21:46:36.693926 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:46:36.693901    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt_odh-ai-gateway-infra(05e7549d-6c49-4fb9-a868-5c94d29f279a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt" podUID="05e7549d-6c49-4fb9-a868-5c94d29f279a"
Jul 17 21:46:37.719065 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:37.719029    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt"]
Jul 17 21:46:37.862137 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:37.862109    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt"
Jul 17 21:46:37.975783 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:37.975687    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e7549d-6c49-4fb9-a868-5c94d29f279a-maas-api-service-ca\") pod \"05e7549d-6c49-4fb9-a868-5c94d29f279a\" (UID: \"05e7549d-6c49-4fb9-a868-5c94d29f279a\") "
Jul 17 21:46:37.976100 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:37.976074    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e7549d-6c49-4fb9-a868-5c94d29f279a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05e7549d-6c49-4fb9-a868-5c94d29f279a" (UID: "05e7549d-6c49-4fb9-a868-5c94d29f279a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:46:38.077287 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:38.077244    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e7549d-6c49-4fb9-a868-5c94d29f279a-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:46:38.706804 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:38.706776    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt"
Jul 17 21:46:38.706978 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:38.706774    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt" event={"ID":"05e7549d-6c49-4fb9-a868-5c94d29f279a","Type":"ContainerDied","Data":"eb335dc36b83c85eadf697422fe27580f62a59ac9892b0f9e1a4e73b26db67fc"}
Jul 17 21:46:38.706978 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:38.706906    2569 scope.go:117] "RemoveContainer" containerID="a6fb4b32c7c300718394b4085464e6c5e601128bff83ed047826367214c5f993"
Jul 17 21:46:38.722478 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:38.722452    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt"]
Jul 17 21:46:38.724695 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:38.724676    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-gqrdt"]
Jul 17 21:46:40.657040 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:40.657006    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05e7549d-6c49-4fb9-a868-5c94d29f279a" path="/var/lib/kubelet/pods/05e7549d-6c49-4fb9-a868-5c94d29f279a/volumes"
Jul 17 21:46:47.266497 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:47.266460    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:49.188715 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:49.188679    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:49.825292 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:49.825245    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:46:52.501084 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:52.501049    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:53.321195 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:53.321135    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:58.880824 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:58.880781    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:58.951014 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:58.950982    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:59.141242 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:46:59.141154    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:00.397945 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:00.397909    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:00.502046 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:00.502013    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:00.617567 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:00.617540    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:00.982526 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:00.982492    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:01.331234 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:01.331199    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:01.430916 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:01.430880    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:07.729942 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:07.729889    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:47:08.804097 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:08.804058    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:08.873610 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:08.873563    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:12.726573 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:12.726536    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:49.118970 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.118925    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w"]
Jul 17 21:47:49.119629 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.119516    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05e7549d-6c49-4fb9-a868-5c94d29f279a" containerName="revoke-keys"
Jul 17 21:47:49.119629 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.119538    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="05e7549d-6c49-4fb9-a868-5c94d29f279a" containerName="revoke-keys"
Jul 17 21:47:49.119629 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.119551    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9cc0744-665e-4c37-a75c-cb96951cc2f0" containerName="revoke-keys"
Jul 17 21:47:49.119629 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.119560    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9cc0744-665e-4c37-a75c-cb96951cc2f0" containerName="revoke-keys"
Jul 17 21:47:49.119629 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.119573    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65150b42-1037-4ec5-92fc-e332d7455706" containerName="cleanup"
Jul 17 21:47:49.119629 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.119583    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="65150b42-1037-4ec5-92fc-e332d7455706" containerName="cleanup"
Jul 17 21:47:49.119629 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.119593    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05e7549d-6c49-4fb9-a868-5c94d29f279a" containerName="revoke-keys"
Jul 17 21:47:49.119629 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.119602    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="05e7549d-6c49-4fb9-a868-5c94d29f279a" containerName="revoke-keys"
Jul 17 21:47:49.119629 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.119627    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05e7549d-6c49-4fb9-a868-5c94d29f279a" containerName="revoke-keys"
Jul 17 21:47:49.119629 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.119634    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="05e7549d-6c49-4fb9-a868-5c94d29f279a" containerName="revoke-keys"
Jul 17 21:47:49.120146 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.119753    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="05e7549d-6c49-4fb9-a868-5c94d29f279a" containerName="revoke-keys"
Jul 17 21:47:49.120146 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.119768    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="05e7549d-6c49-4fb9-a868-5c94d29f279a" containerName="revoke-keys"
Jul 17 21:47:49.120146 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.119781    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="05e7549d-6c49-4fb9-a868-5c94d29f279a" containerName="revoke-keys"
Jul 17 21:47:49.120146 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.119791    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="65150b42-1037-4ec5-92fc-e332d7455706" containerName="cleanup"
Jul 17 21:47:49.123126 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.123103    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w"
Jul 17 21:47:49.125185 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.125166    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:47:49.125333 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.125313    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:47:49.129230 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.129205    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w"]
Jul 17 21:47:49.212030 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.211991    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1e4bf8-d59a-416e-8d4c-7be67ba393f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w\" (UID: \"2d1e4bf8-d59a-416e-8d4c-7be67ba393f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w"
Jul 17 21:47:49.313284 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.313238    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1e4bf8-d59a-416e-8d4c-7be67ba393f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w\" (UID: \"2d1e4bf8-d59a-416e-8d4c-7be67ba393f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w"
Jul 17 21:47:49.313915 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.313892    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1e4bf8-d59a-416e-8d4c-7be67ba393f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w\" (UID: \"2d1e4bf8-d59a-416e-8d4c-7be67ba393f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w"
Jul 17 21:47:49.437531 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.437433    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w"
Jul 17 21:47:49.576544 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:49.576515    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w"]
Jul 17 21:47:49.578482 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:47:49.578449    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1e4bf8_d59a_416e_8d4c_7be67ba393f4.slice/crio-4bbe023272a2b951ceaa27dde768b8caaa8ae020014c878cc270fe87b66e831d WatchSource:0}: Error finding container 4bbe023272a2b951ceaa27dde768b8caaa8ae020014c878cc270fe87b66e831d: Status 404 returned error can't find the container with id 4bbe023272a2b951ceaa27dde768b8caaa8ae020014c878cc270fe87b66e831d
Jul 17 21:47:50.107875 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:50.107840    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w" event={"ID":"2d1e4bf8-d59a-416e-8d4c-7be67ba393f4","Type":"ContainerStarted","Data":"ca116807d9a69f4f42b3a44b794a8cc32ff0b471ed919a1e8d2119e0f2186601"}
Jul 17 21:47:50.108056 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:50.107900    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w" event={"ID":"2d1e4bf8-d59a-416e-8d4c-7be67ba393f4","Type":"ContainerStarted","Data":"4bbe023272a2b951ceaa27dde768b8caaa8ae020014c878cc270fe87b66e831d"}
Jul 17 21:47:50.124570 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:50.124511    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w" podStartSLOduration=1.124495863 podStartE2EDuration="1.124495863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:50.123051541 +0000 UTC m=+2383.882618238" watchObservedRunningTime="2026-07-17 21:47:50.124495863 +0000 UTC m=+2383.884062553"
Jul 17 21:47:51.114754 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:51.114720    2569 generic.go:356] "Generic (PLEG): container finished" podID="2d1e4bf8-d59a-416e-8d4c-7be67ba393f4" containerID="ca116807d9a69f4f42b3a44b794a8cc32ff0b471ed919a1e8d2119e0f2186601" exitCode=6
Jul 17 21:47:51.114931 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:51.114804    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w" event={"ID":"2d1e4bf8-d59a-416e-8d4c-7be67ba393f4","Type":"ContainerDied","Data":"ca116807d9a69f4f42b3a44b794a8cc32ff0b471ed919a1e8d2119e0f2186601"}
Jul 17 21:47:51.114976 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:51.114964    2569 scope.go:117] "RemoveContainer" containerID="ca116807d9a69f4f42b3a44b794a8cc32ff0b471ed919a1e8d2119e0f2186601"
Jul 17 21:47:52.122797 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:52.122699    2569 generic.go:356] "Generic (PLEG): container finished" podID="2d1e4bf8-d59a-416e-8d4c-7be67ba393f4" containerID="f0e2843c0b624cb0f0a0e4f26f068a554db83cd77e2a6519726f1342d5830a66" exitCode=6
Jul 17 21:47:52.123241 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:52.122794    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w" event={"ID":"2d1e4bf8-d59a-416e-8d4c-7be67ba393f4","Type":"ContainerDied","Data":"f0e2843c0b624cb0f0a0e4f26f068a554db83cd77e2a6519726f1342d5830a66"}
Jul 17 21:47:52.123241 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:52.122865    2569 scope.go:117] "RemoveContainer" containerID="ca116807d9a69f4f42b3a44b794a8cc32ff0b471ed919a1e8d2119e0f2186601"
Jul 17 21:47:52.123241 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:52.123012    2569 scope.go:117] "RemoveContainer" containerID="f0e2843c0b624cb0f0a0e4f26f068a554db83cd77e2a6519726f1342d5830a66"
Jul 17 21:47:52.123241 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:47:52.123223    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w_odh-ai-gateway-infra(2d1e4bf8-d59a-416e-8d4c-7be67ba393f4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w" podUID="2d1e4bf8-d59a-416e-8d4c-7be67ba393f4"
Jul 17 21:47:53.130534 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:47:53.130503    2569 scope.go:117] "RemoveContainer" containerID="f0e2843c0b624cb0f0a0e4f26f068a554db83cd77e2a6519726f1342d5830a66"
Jul 17 21:47:53.130913 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:47:53.130691    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w_odh-ai-gateway-infra(2d1e4bf8-d59a-416e-8d4c-7be67ba393f4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w" podUID="2d1e4bf8-d59a-416e-8d4c-7be67ba393f4"
Jul 17 21:48:05.650266 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:05.650228    2569 scope.go:117] "RemoveContainer" containerID="f0e2843c0b624cb0f0a0e4f26f068a554db83cd77e2a6519726f1342d5830a66"
Jul 17 21:48:06.202004 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:06.201971    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w" event={"ID":"2d1e4bf8-d59a-416e-8d4c-7be67ba393f4","Type":"ContainerStarted","Data":"d20197860e91868add899acad6a575f77426028bb47768e2ab38ad5f7aee9c8c"}
Jul 17 21:48:06.825107 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:06.825082    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:48:06.853890 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:06.853864    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:48:07.210272 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:07.210238    2569 generic.go:356] "Generic (PLEG): container finished" podID="2d1e4bf8-d59a-416e-8d4c-7be67ba393f4" containerID="d20197860e91868add899acad6a575f77426028bb47768e2ab38ad5f7aee9c8c" exitCode=6
Jul 17 21:48:07.210476 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:07.210311    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w" event={"ID":"2d1e4bf8-d59a-416e-8d4c-7be67ba393f4","Type":"ContainerDied","Data":"d20197860e91868add899acad6a575f77426028bb47768e2ab38ad5f7aee9c8c"}
Jul 17 21:48:07.210476 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:07.210397    2569 scope.go:117] "RemoveContainer" containerID="f0e2843c0b624cb0f0a0e4f26f068a554db83cd77e2a6519726f1342d5830a66"
Jul 17 21:48:07.210661 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:07.210648    2569 scope.go:117] "RemoveContainer" containerID="d20197860e91868add899acad6a575f77426028bb47768e2ab38ad5f7aee9c8c"
Jul 17 21:48:07.210881 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:48:07.210863    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w_odh-ai-gateway-infra(2d1e4bf8-d59a-416e-8d4c-7be67ba393f4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w" podUID="2d1e4bf8-d59a-416e-8d4c-7be67ba393f4"
Jul 17 21:48:07.228591 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:07.228557    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w"]
Jul 17 21:48:08.359304 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:08.359278    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w"
Jul 17 21:48:08.377625 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:08.377591    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1e4bf8-d59a-416e-8d4c-7be67ba393f4-maas-api-service-ca\") pod \"2d1e4bf8-d59a-416e-8d4c-7be67ba393f4\" (UID: \"2d1e4bf8-d59a-416e-8d4c-7be67ba393f4\") "
Jul 17 21:48:08.377994 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:08.377972    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1e4bf8-d59a-416e-8d4c-7be67ba393f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d1e4bf8-d59a-416e-8d4c-7be67ba393f4" (UID: "2d1e4bf8-d59a-416e-8d4c-7be67ba393f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:48:08.478771 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:08.478689    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1e4bf8-d59a-416e-8d4c-7be67ba393f4-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:48:09.224641 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:09.224597    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w" event={"ID":"2d1e4bf8-d59a-416e-8d4c-7be67ba393f4","Type":"ContainerDied","Data":"4bbe023272a2b951ceaa27dde768b8caaa8ae020014c878cc270fe87b66e831d"}
Jul 17 21:48:09.224844 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:09.224664    2569 scope.go:117] "RemoveContainer" containerID="d20197860e91868add899acad6a575f77426028bb47768e2ab38ad5f7aee9c8c"
Jul 17 21:48:09.224844 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:09.224612    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w"
Jul 17 21:48:09.239358 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:09.239329    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w"]
Jul 17 21:48:09.247169 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:09.247140    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9xg2w"]
Jul 17 21:48:10.324951 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:10.324910    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp"]
Jul 17 21:48:10.325544 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:10.325524    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1e4bf8-d59a-416e-8d4c-7be67ba393f4" containerName="revoke-keys"
Jul 17 21:48:10.325648 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:10.325547    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1e4bf8-d59a-416e-8d4c-7be67ba393f4" containerName="revoke-keys"
Jul 17 21:48:10.325648 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:10.325572    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1e4bf8-d59a-416e-8d4c-7be67ba393f4" containerName="revoke-keys"
Jul 17 21:48:10.325648 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:10.325581    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1e4bf8-d59a-416e-8d4c-7be67ba393f4" containerName="revoke-keys"
Jul 17 21:48:10.325801 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:10.325690    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1e4bf8-d59a-416e-8d4c-7be67ba393f4" containerName="revoke-keys"
Jul 17 21:48:10.325801 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:10.325702    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1e4bf8-d59a-416e-8d4c-7be67ba393f4" containerName="revoke-keys"
Jul 17 21:48:10.325801 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:10.325712    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1e4bf8-d59a-416e-8d4c-7be67ba393f4" containerName="revoke-keys"
Jul 17 21:48:10.330130 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:10.330108    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp"
Jul 17 21:48:10.332432 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:10.332410    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:48:10.332569 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:10.332548    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:48:10.336952 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:10.336928    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp"]
Jul 17 21:48:10.394561 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:10.394521    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61d4787-6388-4b7c-b40d-142bb3270ebf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-4jhzp\" (UID: \"c61d4787-6388-4b7c-b40d-142bb3270ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp"
Jul 17 21:48:10.495417 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:10.495372    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61d4787-6388-4b7c-b40d-142bb3270ebf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-4jhzp\" (UID: \"c61d4787-6388-4b7c-b40d-142bb3270ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp"
Jul 17 21:48:10.496096 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:10.496076    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61d4787-6388-4b7c-b40d-142bb3270ebf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-4jhzp\" (UID: \"c61d4787-6388-4b7c-b40d-142bb3270ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp"
Jul 17 21:48:10.642506 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:10.642463    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp"
Jul 17 21:48:10.657594 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:10.657556    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d1e4bf8-d59a-416e-8d4c-7be67ba393f4" path="/var/lib/kubelet/pods/2d1e4bf8-d59a-416e-8d4c-7be67ba393f4/volumes"
Jul 17 21:48:10.782008 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:10.781986    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp"]
Jul 17 21:48:10.784028 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:48:10.783999    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61d4787_6388_4b7c_b40d_142bb3270ebf.slice/crio-0ffc08ec91824b8df74353cf6fb07ea29c7f9680fe2e2a08f452bbd0c2ae233d WatchSource:0}: Error finding container 0ffc08ec91824b8df74353cf6fb07ea29c7f9680fe2e2a08f452bbd0c2ae233d: Status 404 returned error can't find the container with id 0ffc08ec91824b8df74353cf6fb07ea29c7f9680fe2e2a08f452bbd0c2ae233d
Jul 17 21:48:11.047348 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:11.047318    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:48:11.241318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:11.241158    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp" event={"ID":"c61d4787-6388-4b7c-b40d-142bb3270ebf","Type":"ContainerStarted","Data":"c53a7f3cc97bce573670d89f2d851a6f6b409cb7645611d0e71eaac26f45d3c3"}
Jul 17 21:48:11.244992 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:11.241630    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp" event={"ID":"c61d4787-6388-4b7c-b40d-142bb3270ebf","Type":"ContainerStarted","Data":"0ffc08ec91824b8df74353cf6fb07ea29c7f9680fe2e2a08f452bbd0c2ae233d"}
Jul 17 21:48:11.258372 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:11.258319    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp" podStartSLOduration=1.258303792 podStartE2EDuration="1.258303792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:48:11.257240973 +0000 UTC m=+2405.016807666" watchObservedRunningTime="2026-07-17 21:48:11.258303792 +0000 UTC m=+2405.017870490"
Jul 17 21:48:12.247846 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:12.247812    2569 generic.go:356] "Generic (PLEG): container finished" podID="c61d4787-6388-4b7c-b40d-142bb3270ebf" containerID="c53a7f3cc97bce573670d89f2d851a6f6b409cb7645611d0e71eaac26f45d3c3" exitCode=6
Jul 17 21:48:12.248252 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:12.247881    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp" event={"ID":"c61d4787-6388-4b7c-b40d-142bb3270ebf","Type":"ContainerDied","Data":"c53a7f3cc97bce573670d89f2d851a6f6b409cb7645611d0e71eaac26f45d3c3"}
Jul 17 21:48:12.248252 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:12.248084    2569 scope.go:117] "RemoveContainer" containerID="c53a7f3cc97bce573670d89f2d851a6f6b409cb7645611d0e71eaac26f45d3c3"
Jul 17 21:48:13.255574 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:13.255486    2569 generic.go:356] "Generic (PLEG): container finished" podID="c61d4787-6388-4b7c-b40d-142bb3270ebf" containerID="624076c0ccd9ae740a7ad00b289fb632de0ed5fc1f9ef860d70da7253271bb78" exitCode=6
Jul 17 21:48:13.255970 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:13.255581    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp" event={"ID":"c61d4787-6388-4b7c-b40d-142bb3270ebf","Type":"ContainerDied","Data":"624076c0ccd9ae740a7ad00b289fb632de0ed5fc1f9ef860d70da7253271bb78"}
Jul 17 21:48:13.255970 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:13.255643    2569 scope.go:117] "RemoveContainer" containerID="c53a7f3cc97bce573670d89f2d851a6f6b409cb7645611d0e71eaac26f45d3c3"
Jul 17 21:48:13.255970 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:13.255797    2569 scope.go:117] "RemoveContainer" containerID="624076c0ccd9ae740a7ad00b289fb632de0ed5fc1f9ef860d70da7253271bb78"
Jul 17 21:48:13.256114 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:48:13.256019    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-4jhzp_odh-ai-gateway-infra(c61d4787-6388-4b7c-b40d-142bb3270ebf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp" podUID="c61d4787-6388-4b7c-b40d-142bb3270ebf"
Jul 17 21:48:14.263109 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:14.263082    2569 scope.go:117] "RemoveContainer" containerID="624076c0ccd9ae740a7ad00b289fb632de0ed5fc1f9ef860d70da7253271bb78"
Jul 17 21:48:14.263533 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:48:14.263270    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-4jhzp_odh-ai-gateway-infra(c61d4787-6388-4b7c-b40d-142bb3270ebf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp" podUID="c61d4787-6388-4b7c-b40d-142bb3270ebf"
Jul 17 21:48:27.650373 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:27.650335    2569 scope.go:117] "RemoveContainer" containerID="624076c0ccd9ae740a7ad00b289fb632de0ed5fc1f9ef860d70da7253271bb78"
Jul 17 21:48:28.340172 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:28.340143    2569 generic.go:356] "Generic (PLEG): container finished" podID="c61d4787-6388-4b7c-b40d-142bb3270ebf" containerID="929b0f4f1a528b24684801f7b6eb96d51ca8caaf486c7c2f95f6a8072d4f1a57" exitCode=6
Jul 17 21:48:28.340362 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:28.340222    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp" event={"ID":"c61d4787-6388-4b7c-b40d-142bb3270ebf","Type":"ContainerDied","Data":"929b0f4f1a528b24684801f7b6eb96d51ca8caaf486c7c2f95f6a8072d4f1a57"}
Jul 17 21:48:28.340362 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:28.340297    2569 scope.go:117] "RemoveContainer" containerID="624076c0ccd9ae740a7ad00b289fb632de0ed5fc1f9ef860d70da7253271bb78"
Jul 17 21:48:28.340655 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:28.340640    2569 scope.go:117] "RemoveContainer" containerID="929b0f4f1a528b24684801f7b6eb96d51ca8caaf486c7c2f95f6a8072d4f1a57"
Jul 17 21:48:28.340874 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:48:28.340854    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-4jhzp_odh-ai-gateway-infra(c61d4787-6388-4b7c-b40d-142bb3270ebf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp" podUID="c61d4787-6388-4b7c-b40d-142bb3270ebf"
Jul 17 21:48:29.367816 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:29.367744    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp"]
Jul 17 21:48:29.506380 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:29.506354    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp"
Jul 17 21:48:29.566007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:29.565968    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61d4787-6388-4b7c-b40d-142bb3270ebf-maas-api-service-ca\") pod \"c61d4787-6388-4b7c-b40d-142bb3270ebf\" (UID: \"c61d4787-6388-4b7c-b40d-142bb3270ebf\") "
Jul 17 21:48:29.566375 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:29.566348    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61d4787-6388-4b7c-b40d-142bb3270ebf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c61d4787-6388-4b7c-b40d-142bb3270ebf" (UID: "c61d4787-6388-4b7c-b40d-142bb3270ebf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:48:29.666645 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:29.666555    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61d4787-6388-4b7c-b40d-142bb3270ebf-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:48:30.353848 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:30.353811    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp" event={"ID":"c61d4787-6388-4b7c-b40d-142bb3270ebf","Type":"ContainerDied","Data":"0ffc08ec91824b8df74353cf6fb07ea29c7f9680fe2e2a08f452bbd0c2ae233d"}
Jul 17 21:48:30.354043 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:30.353878    2569 scope.go:117] "RemoveContainer" containerID="929b0f4f1a528b24684801f7b6eb96d51ca8caaf486c7c2f95f6a8072d4f1a57"
Jul 17 21:48:30.354043 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:30.353824    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp"
Jul 17 21:48:30.380241 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:30.380213    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp"]
Jul 17 21:48:30.385159 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:30.385134    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-4jhzp"]
Jul 17 21:48:30.656286 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:30.656205    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c61d4787-6388-4b7c-b40d-142bb3270ebf" path="/var/lib/kubelet/pods/c61d4787-6388-4b7c-b40d-142bb3270ebf/volumes"
Jul 17 21:48:44.748960 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:44.748925    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r"]
Jul 17 21:48:44.749342 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:44.749311    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c61d4787-6388-4b7c-b40d-142bb3270ebf" containerName="revoke-keys"
Jul 17 21:48:44.749342 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:44.749321    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c61d4787-6388-4b7c-b40d-142bb3270ebf" containerName="revoke-keys"
Jul 17 21:48:44.749342 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:44.749330    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1e4bf8-d59a-416e-8d4c-7be67ba393f4" containerName="revoke-keys"
Jul 17 21:48:44.749342 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:44.749335    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1e4bf8-d59a-416e-8d4c-7be67ba393f4" containerName="revoke-keys"
Jul 17 21:48:44.749342 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:44.749343    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c61d4787-6388-4b7c-b40d-142bb3270ebf" containerName="revoke-keys"
Jul 17 21:48:44.749531 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:44.749350    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c61d4787-6388-4b7c-b40d-142bb3270ebf" containerName="revoke-keys"
Jul 17 21:48:44.749531 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:44.749358    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c61d4787-6388-4b7c-b40d-142bb3270ebf" containerName="revoke-keys"
Jul 17 21:48:44.749531 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:44.749365    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c61d4787-6388-4b7c-b40d-142bb3270ebf" containerName="revoke-keys"
Jul 17 21:48:44.749531 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:44.749441    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c61d4787-6388-4b7c-b40d-142bb3270ebf" containerName="revoke-keys"
Jul 17 21:48:44.749531 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:44.749448    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c61d4787-6388-4b7c-b40d-142bb3270ebf" containerName="revoke-keys"
Jul 17 21:48:44.749531 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:44.749456    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c61d4787-6388-4b7c-b40d-142bb3270ebf" containerName="revoke-keys"
Jul 17 21:48:44.752467 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:44.752450    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r"
Jul 17 21:48:44.754506 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:44.754480    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:48:44.754625 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:44.754548    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:48:44.760370 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:44.760348    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r"]
Jul 17 21:48:44.900386 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:44.900351    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc286d91-467e-4d8f-aa7f-6b41b51a0bc5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r\" (UID: \"fc286d91-467e-4d8f-aa7f-6b41b51a0bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r"
Jul 17 21:48:45.001694 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:45.001599    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc286d91-467e-4d8f-aa7f-6b41b51a0bc5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r\" (UID: \"fc286d91-467e-4d8f-aa7f-6b41b51a0bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r"
Jul 17 21:48:45.002237 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:45.002217    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc286d91-467e-4d8f-aa7f-6b41b51a0bc5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r\" (UID: \"fc286d91-467e-4d8f-aa7f-6b41b51a0bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r"
Jul 17 21:48:45.064247 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:45.064212    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r"
Jul 17 21:48:45.198452 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:45.198427    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r"]
Jul 17 21:48:45.201231 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:48:45.201194    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc286d91_467e_4d8f_aa7f_6b41b51a0bc5.slice/crio-1fad3cb46860032ae1f53beef0fddff7b8672327503d1177a6f6338bd8853ae9 WatchSource:0}: Error finding container 1fad3cb46860032ae1f53beef0fddff7b8672327503d1177a6f6338bd8853ae9: Status 404 returned error can't find the container with id 1fad3cb46860032ae1f53beef0fddff7b8672327503d1177a6f6338bd8853ae9
Jul 17 21:48:45.444045 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:45.444006    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r" event={"ID":"fc286d91-467e-4d8f-aa7f-6b41b51a0bc5","Type":"ContainerStarted","Data":"1fad3cb46860032ae1f53beef0fddff7b8672327503d1177a6f6338bd8853ae9"}
Jul 17 21:48:46.451384 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:46.451343    2569 generic.go:356] "Generic (PLEG): container finished" podID="fc286d91-467e-4d8f-aa7f-6b41b51a0bc5" containerID="5060e8b8ee18b4115ca0d4b5dd443a0d8d56c72dc7aa597465ad4a297ef4c89f" exitCode=7
Jul 17 21:48:46.451834 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:46.451433    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r" event={"ID":"fc286d91-467e-4d8f-aa7f-6b41b51a0bc5","Type":"ContainerDied","Data":"5060e8b8ee18b4115ca0d4b5dd443a0d8d56c72dc7aa597465ad4a297ef4c89f"}
Jul 17 21:48:46.451834 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:46.451684    2569 scope.go:117] "RemoveContainer" containerID="5060e8b8ee18b4115ca0d4b5dd443a0d8d56c72dc7aa597465ad4a297ef4c89f"
Jul 17 21:48:47.459092 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:47.459062    2569 generic.go:356] "Generic (PLEG): container finished" podID="fc286d91-467e-4d8f-aa7f-6b41b51a0bc5" containerID="021150ee79cb04b7fa71e330bbd65665c81f6f8f1a79d2e99cd42bbce040ddbb" exitCode=7
Jul 17 21:48:47.459598 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:47.459152    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r" event={"ID":"fc286d91-467e-4d8f-aa7f-6b41b51a0bc5","Type":"ContainerDied","Data":"021150ee79cb04b7fa71e330bbd65665c81f6f8f1a79d2e99cd42bbce040ddbb"}
Jul 17 21:48:47.459598 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:47.459219    2569 scope.go:117] "RemoveContainer" containerID="5060e8b8ee18b4115ca0d4b5dd443a0d8d56c72dc7aa597465ad4a297ef4c89f"
Jul 17 21:48:47.459598 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:47.459363    2569 scope.go:117] "RemoveContainer" containerID="021150ee79cb04b7fa71e330bbd65665c81f6f8f1a79d2e99cd42bbce040ddbb"
Jul 17 21:48:47.459776 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:48:47.459610    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r_odh-ai-gateway-infra(fc286d91-467e-4d8f-aa7f-6b41b51a0bc5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r" podUID="fc286d91-467e-4d8f-aa7f-6b41b51a0bc5"
Jul 17 21:48:48.467128 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:48.467101    2569 scope.go:117] "RemoveContainer" containerID="021150ee79cb04b7fa71e330bbd65665c81f6f8f1a79d2e99cd42bbce040ddbb"
Jul 17 21:48:48.467526 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:48:48.467278    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r_odh-ai-gateway-infra(fc286d91-467e-4d8f-aa7f-6b41b51a0bc5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r" podUID="fc286d91-467e-4d8f-aa7f-6b41b51a0bc5"
Jul 17 21:48:49.171989 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:49.171951    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28"]
Jul 17 21:48:49.176751 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:49.176733    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28"
Jul 17 21:48:49.183758 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:49.183482    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28"]
Jul 17 21:48:49.338877 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:49.338833    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cd346b-d8b0-4dff-9f90-3975660a0a3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28\" (UID: \"f3cd346b-d8b0-4dff-9f90-3975660a0a3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28"
Jul 17 21:48:49.439815 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:49.439722    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cd346b-d8b0-4dff-9f90-3975660a0a3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28\" (UID: \"f3cd346b-d8b0-4dff-9f90-3975660a0a3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28"
Jul 17 21:48:49.440363 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:49.440341    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cd346b-d8b0-4dff-9f90-3975660a0a3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28\" (UID: \"f3cd346b-d8b0-4dff-9f90-3975660a0a3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28"
Jul 17 21:48:49.489371 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:49.489344    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28"
Jul 17 21:48:49.627584 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:49.627549    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28"]
Jul 17 21:48:49.629346 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:48:49.629311    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3cd346b_d8b0_4dff_9f90_3975660a0a3c.slice/crio-f289c1337623ae61698f6f913c3e02ba3d18ce799144bfec6af9db76723f3869 WatchSource:0}: Error finding container f289c1337623ae61698f6f913c3e02ba3d18ce799144bfec6af9db76723f3869: Status 404 returned error can't find the container with id f289c1337623ae61698f6f913c3e02ba3d18ce799144bfec6af9db76723f3869
Jul 17 21:48:50.480691 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:50.480656    2569 generic.go:356] "Generic (PLEG): container finished" podID="f3cd346b-d8b0-4dff-9f90-3975660a0a3c" containerID="fd11841e0cfba204c0c85b057037afe123ec44083fd3658879ae7be128b3bf6d" exitCode=6
Jul 17 21:48:50.480891 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:50.480718    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28" event={"ID":"f3cd346b-d8b0-4dff-9f90-3975660a0a3c","Type":"ContainerDied","Data":"fd11841e0cfba204c0c85b057037afe123ec44083fd3658879ae7be128b3bf6d"}
Jul 17 21:48:50.480891 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:50.480767    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28" event={"ID":"f3cd346b-d8b0-4dff-9f90-3975660a0a3c","Type":"ContainerStarted","Data":"f289c1337623ae61698f6f913c3e02ba3d18ce799144bfec6af9db76723f3869"}
Jul 17 21:48:50.481011 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:50.481000    2569 scope.go:117] "RemoveContainer" containerID="fd11841e0cfba204c0c85b057037afe123ec44083fd3658879ae7be128b3bf6d"
Jul 17 21:48:51.110846 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:48:51.110813    2569 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3cd346b_d8b0_4dff_9f90_3975660a0a3c.slice/crio-conmon-2f8f276f40e3de31a518a8699fd824828c292b1d2f211cc71f5ba20031c816f1.scope\": RecentStats: unable to find data in memory cache]"
Jul 17 21:48:51.489271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:51.489234    2569 generic.go:356] "Generic (PLEG): container finished" podID="f3cd346b-d8b0-4dff-9f90-3975660a0a3c" containerID="2f8f276f40e3de31a518a8699fd824828c292b1d2f211cc71f5ba20031c816f1" exitCode=6
Jul 17 21:48:51.489496 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:51.489318    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28" event={"ID":"f3cd346b-d8b0-4dff-9f90-3975660a0a3c","Type":"ContainerDied","Data":"2f8f276f40e3de31a518a8699fd824828c292b1d2f211cc71f5ba20031c816f1"}
Jul 17 21:48:51.489496 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:51.489375    2569 scope.go:117] "RemoveContainer" containerID="fd11841e0cfba204c0c85b057037afe123ec44083fd3658879ae7be128b3bf6d"
Jul 17 21:48:51.489606 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:51.489562    2569 scope.go:117] "RemoveContainer" containerID="2f8f276f40e3de31a518a8699fd824828c292b1d2f211cc71f5ba20031c816f1"
Jul 17 21:48:51.489816 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:48:51.489796    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28_odh-ai-gateway-infra(f3cd346b-d8b0-4dff-9f90-3975660a0a3c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28" podUID="f3cd346b-d8b0-4dff-9f90-3975660a0a3c"
Jul 17 21:48:52.504177 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:52.504151    2569 scope.go:117] "RemoveContainer" containerID="2f8f276f40e3de31a518a8699fd824828c292b1d2f211cc71f5ba20031c816f1"
Jul 17 21:48:52.504637 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:48:52.504389    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28_odh-ai-gateway-infra(f3cd346b-d8b0-4dff-9f90-3975660a0a3c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28" podUID="f3cd346b-d8b0-4dff-9f90-3975660a0a3c"
Jul 17 21:48:58.650527 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:58.650496    2569 scope.go:117] "RemoveContainer" containerID="021150ee79cb04b7fa71e330bbd65665c81f6f8f1a79d2e99cd42bbce040ddbb"
Jul 17 21:48:59.545762 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:59.545727    2569 generic.go:356] "Generic (PLEG): container finished" podID="fc286d91-467e-4d8f-aa7f-6b41b51a0bc5" containerID="a90826deadd5954cf4de2c0f499c0c4b4e96947f3b34290a8bbb8d546b47f797" exitCode=7
Jul 17 21:48:59.545960 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:59.545810    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r" event={"ID":"fc286d91-467e-4d8f-aa7f-6b41b51a0bc5","Type":"ContainerDied","Data":"a90826deadd5954cf4de2c0f499c0c4b4e96947f3b34290a8bbb8d546b47f797"}
Jul 17 21:48:59.545960 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:59.545891    2569 scope.go:117] "RemoveContainer" containerID="021150ee79cb04b7fa71e330bbd65665c81f6f8f1a79d2e99cd42bbce040ddbb"
Jul 17 21:48:59.546202 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:48:59.546186    2569 scope.go:117] "RemoveContainer" containerID="a90826deadd5954cf4de2c0f499c0c4b4e96947f3b34290a8bbb8d546b47f797"
Jul 17 21:48:59.547497 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:48:59.547476    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r_odh-ai-gateway-infra(fc286d91-467e-4d8f-aa7f-6b41b51a0bc5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r" podUID="fc286d91-467e-4d8f-aa7f-6b41b51a0bc5"
Jul 17 21:49:00.572415 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:00.572366    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r"]
Jul 17 21:49:00.709658 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:00.709626    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r"
Jul 17 21:49:00.741979 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:00.741947    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc286d91-467e-4d8f-aa7f-6b41b51a0bc5-maas-api-service-ca\") pod \"fc286d91-467e-4d8f-aa7f-6b41b51a0bc5\" (UID: \"fc286d91-467e-4d8f-aa7f-6b41b51a0bc5\") "
Jul 17 21:49:00.742315 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:00.742294    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc286d91-467e-4d8f-aa7f-6b41b51a0bc5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fc286d91-467e-4d8f-aa7f-6b41b51a0bc5" (UID: "fc286d91-467e-4d8f-aa7f-6b41b51a0bc5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:49:00.843054 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:00.843019    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc286d91-467e-4d8f-aa7f-6b41b51a0bc5-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:49:01.366532 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:01.366499    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:01.467245 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:01.467212    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:01.560081 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:01.560052    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r"
Jul 17 21:49:01.560081 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:01.560063    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r" event={"ID":"fc286d91-467e-4d8f-aa7f-6b41b51a0bc5","Type":"ContainerDied","Data":"1fad3cb46860032ae1f53beef0fddff7b8672327503d1177a6f6338bd8853ae9"}
Jul 17 21:49:01.560333 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:01.560125    2569 scope.go:117] "RemoveContainer" containerID="a90826deadd5954cf4de2c0f499c0c4b4e96947f3b34290a8bbb8d546b47f797"
Jul 17 21:49:01.597858 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:01.597815    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r"]
Jul 17 21:49:01.599986 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:01.599966    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-dlh6r"]
Jul 17 21:49:02.656553 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:02.656519    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc286d91-467e-4d8f-aa7f-6b41b51a0bc5" path="/var/lib/kubelet/pods/fc286d91-467e-4d8f-aa7f-6b41b51a0bc5/volumes"
Jul 17 21:49:03.650804 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:03.650766    2569 scope.go:117] "RemoveContainer" containerID="2f8f276f40e3de31a518a8699fd824828c292b1d2f211cc71f5ba20031c816f1"
Jul 17 21:49:04.584251 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:04.584214    2569 generic.go:356] "Generic (PLEG): container finished" podID="f3cd346b-d8b0-4dff-9f90-3975660a0a3c" containerID="70a4fd8adc0d52248a19a9116fad2cf7ea97391c1ec325f9c6769fab15b81bb2" exitCode=6
Jul 17 21:49:04.584752 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:04.584287    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28" event={"ID":"f3cd346b-d8b0-4dff-9f90-3975660a0a3c","Type":"ContainerDied","Data":"70a4fd8adc0d52248a19a9116fad2cf7ea97391c1ec325f9c6769fab15b81bb2"}
Jul 17 21:49:04.584752 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:04.584375    2569 scope.go:117] "RemoveContainer" containerID="2f8f276f40e3de31a518a8699fd824828c292b1d2f211cc71f5ba20031c816f1"
Jul 17 21:49:04.584752 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:04.584663    2569 scope.go:117] "RemoveContainer" containerID="70a4fd8adc0d52248a19a9116fad2cf7ea97391c1ec325f9c6769fab15b81bb2"
Jul 17 21:49:04.584901 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:49:04.584881    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28_odh-ai-gateway-infra(f3cd346b-d8b0-4dff-9f90-3975660a0a3c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28" podUID="f3cd346b-d8b0-4dff-9f90-3975660a0a3c"
Jul 17 21:49:04.674381 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:04.674349    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:04.774478 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:04.774443    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:05.608489 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:05.608455    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28"]
Jul 17 21:49:05.738946 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:05.738919    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28"
Jul 17 21:49:05.787190 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:05.787155    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cd346b-d8b0-4dff-9f90-3975660a0a3c-maas-api-service-ca\") pod \"f3cd346b-d8b0-4dff-9f90-3975660a0a3c\" (UID: \"f3cd346b-d8b0-4dff-9f90-3975660a0a3c\") "
Jul 17 21:49:05.787564 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:05.787540    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3cd346b-d8b0-4dff-9f90-3975660a0a3c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f3cd346b-d8b0-4dff-9f90-3975660a0a3c" (UID: "f3cd346b-d8b0-4dff-9f90-3975660a0a3c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:49:05.888204 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:05.888121    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3cd346b-d8b0-4dff-9f90-3975660a0a3c-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:49:06.599196 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:06.599165    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28" event={"ID":"f3cd346b-d8b0-4dff-9f90-3975660a0a3c","Type":"ContainerDied","Data":"f289c1337623ae61698f6f913c3e02ba3d18ce799144bfec6af9db76723f3869"}
Jul 17 21:49:06.599196 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:06.599184    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28"
Jul 17 21:49:06.599473 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:06.599225    2569 scope.go:117] "RemoveContainer" containerID="70a4fd8adc0d52248a19a9116fad2cf7ea97391c1ec325f9c6769fab15b81bb2"
Jul 17 21:49:06.621376 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:06.621347    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28"]
Jul 17 21:49:06.627592 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:06.627564    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jmh28"]
Jul 17 21:49:06.656867 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:06.656836    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3cd346b-d8b0-4dff-9f90-3975660a0a3c" path="/var/lib/kubelet/pods/f3cd346b-d8b0-4dff-9f90-3975660a0a3c/volumes"
Jul 17 21:49:10.373296 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.373250    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j"]
Jul 17 21:49:10.373905 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.373884    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3cd346b-d8b0-4dff-9f90-3975660a0a3c" containerName="revoke-keys"
Jul 17 21:49:10.373992 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.373908    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3cd346b-d8b0-4dff-9f90-3975660a0a3c" containerName="revoke-keys"
Jul 17 21:49:10.373992 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.373945    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc286d91-467e-4d8f-aa7f-6b41b51a0bc5" containerName="revoke-keys"
Jul 17 21:49:10.373992 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.373955    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc286d91-467e-4d8f-aa7f-6b41b51a0bc5" containerName="revoke-keys"
Jul 17 21:49:10.373992 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.373973    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc286d91-467e-4d8f-aa7f-6b41b51a0bc5" containerName="revoke-keys"
Jul 17 21:49:10.373992 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.373984    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc286d91-467e-4d8f-aa7f-6b41b51a0bc5" containerName="revoke-keys"
Jul 17 21:49:10.374236 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.374001    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc286d91-467e-4d8f-aa7f-6b41b51a0bc5" containerName="revoke-keys"
Jul 17 21:49:10.374236 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.374009    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc286d91-467e-4d8f-aa7f-6b41b51a0bc5" containerName="revoke-keys"
Jul 17 21:49:10.374236 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.374020    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3cd346b-d8b0-4dff-9f90-3975660a0a3c" containerName="revoke-keys"
Jul 17 21:49:10.374236 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.374028    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3cd346b-d8b0-4dff-9f90-3975660a0a3c" containerName="revoke-keys"
Jul 17 21:49:10.374236 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.374127    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc286d91-467e-4d8f-aa7f-6b41b51a0bc5" containerName="revoke-keys"
Jul 17 21:49:10.374236 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.374139    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3cd346b-d8b0-4dff-9f90-3975660a0a3c" containerName="revoke-keys"
Jul 17 21:49:10.374236 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.374151    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc286d91-467e-4d8f-aa7f-6b41b51a0bc5" containerName="revoke-keys"
Jul 17 21:49:10.374236 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.374160    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3cd346b-d8b0-4dff-9f90-3975660a0a3c" containerName="revoke-keys"
Jul 17 21:49:10.374236 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.374170    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3cd346b-d8b0-4dff-9f90-3975660a0a3c" containerName="revoke-keys"
Jul 17 21:49:10.378799 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.378777    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j"
Jul 17 21:49:10.380890 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.380871    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:49:10.380890 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.380880    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:49:10.385826 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.385801    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j"]
Jul 17 21:49:10.429010 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.428970    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbdf075-1ce2-482f-afed-d9196a853e2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-8jg6j\" (UID: \"5cbdf075-1ce2-482f-afed-d9196a853e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j"
Jul 17 21:49:10.529797 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.529756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbdf075-1ce2-482f-afed-d9196a853e2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-8jg6j\" (UID: \"5cbdf075-1ce2-482f-afed-d9196a853e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j"
Jul 17 21:49:10.530430 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.530388    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbdf075-1ce2-482f-afed-d9196a853e2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-8jg6j\" (UID: \"5cbdf075-1ce2-482f-afed-d9196a853e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j"
Jul 17 21:49:10.692259 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.692165    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j"
Jul 17 21:49:10.829459 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:10.829429    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j"]
Jul 17 21:49:10.830781 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:49:10.830759    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbdf075_1ce2_482f_afed_d9196a853e2f.slice/crio-1580e9368ea3e905d990b262ef621ceb3dd2576d6cc9bc5da5087936ab38c708 WatchSource:0}: Error finding container 1580e9368ea3e905d990b262ef621ceb3dd2576d6cc9bc5da5087936ab38c708: Status 404 returned error can't find the container with id 1580e9368ea3e905d990b262ef621ceb3dd2576d6cc9bc5da5087936ab38c708
Jul 17 21:49:11.631126 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:11.631087    2569 generic.go:356] "Generic (PLEG): container finished" podID="5cbdf075-1ce2-482f-afed-d9196a853e2f" containerID="7709d2b302f1c053107a2602c11280682c56e9cc499944b2f24526a768705da2" exitCode=6
Jul 17 21:49:11.631580 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:11.631176    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j" event={"ID":"5cbdf075-1ce2-482f-afed-d9196a853e2f","Type":"ContainerDied","Data":"7709d2b302f1c053107a2602c11280682c56e9cc499944b2f24526a768705da2"}
Jul 17 21:49:11.631580 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:11.631244    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j" event={"ID":"5cbdf075-1ce2-482f-afed-d9196a853e2f","Type":"ContainerStarted","Data":"1580e9368ea3e905d990b262ef621ceb3dd2576d6cc9bc5da5087936ab38c708"}
Jul 17 21:49:11.631580 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:11.631379    2569 scope.go:117] "RemoveContainer" containerID="7709d2b302f1c053107a2602c11280682c56e9cc499944b2f24526a768705da2"
Jul 17 21:49:12.639143 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:12.639104    2569 generic.go:356] "Generic (PLEG): container finished" podID="5cbdf075-1ce2-482f-afed-d9196a853e2f" containerID="a785800f401ad41f7ccbc8eb07bc79f88ed1880a3bc501a962fea81f583cda31" exitCode=6
Jul 17 21:49:12.639605 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:12.639170    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j" event={"ID":"5cbdf075-1ce2-482f-afed-d9196a853e2f","Type":"ContainerDied","Data":"a785800f401ad41f7ccbc8eb07bc79f88ed1880a3bc501a962fea81f583cda31"}
Jul 17 21:49:12.639605 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:12.639235    2569 scope.go:117] "RemoveContainer" containerID="7709d2b302f1c053107a2602c11280682c56e9cc499944b2f24526a768705da2"
Jul 17 21:49:12.639605 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:12.639396    2569 scope.go:117] "RemoveContainer" containerID="a785800f401ad41f7ccbc8eb07bc79f88ed1880a3bc501a962fea81f583cda31"
Jul 17 21:49:12.639755 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:49:12.639635    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-8jg6j_odh-ai-gateway-infra(5cbdf075-1ce2-482f-afed-d9196a853e2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j" podUID="5cbdf075-1ce2-482f-afed-d9196a853e2f"
Jul 17 21:49:13.646607 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:13.646578    2569 scope.go:117] "RemoveContainer" containerID="a785800f401ad41f7ccbc8eb07bc79f88ed1880a3bc501a962fea81f583cda31"
Jul 17 21:49:13.646974 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:49:13.646802    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-8jg6j_odh-ai-gateway-infra(5cbdf075-1ce2-482f-afed-d9196a853e2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j" podUID="5cbdf075-1ce2-482f-afed-d9196a853e2f"
Jul 17 21:49:25.650611 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:25.650579    2569 scope.go:117] "RemoveContainer" containerID="a785800f401ad41f7ccbc8eb07bc79f88ed1880a3bc501a962fea81f583cda31"
Jul 17 21:49:26.719839 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:26.719751    2569 generic.go:356] "Generic (PLEG): container finished" podID="5cbdf075-1ce2-482f-afed-d9196a853e2f" containerID="2c3ba40ae8be3dfff107ec40e5df9041b7c4b18a8242a1d9f95b028087bebdb5" exitCode=6
Jul 17 21:49:26.720276 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:26.719830    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j" event={"ID":"5cbdf075-1ce2-482f-afed-d9196a853e2f","Type":"ContainerDied","Data":"2c3ba40ae8be3dfff107ec40e5df9041b7c4b18a8242a1d9f95b028087bebdb5"}
Jul 17 21:49:26.720276 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:26.719902    2569 scope.go:117] "RemoveContainer" containerID="a785800f401ad41f7ccbc8eb07bc79f88ed1880a3bc501a962fea81f583cda31"
Jul 17 21:49:26.720276 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:26.720180    2569 scope.go:117] "RemoveContainer" containerID="2c3ba40ae8be3dfff107ec40e5df9041b7c4b18a8242a1d9f95b028087bebdb5"
Jul 17 21:49:26.720442 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:49:26.720365    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-8jg6j_odh-ai-gateway-infra(5cbdf075-1ce2-482f-afed-d9196a853e2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j" podUID="5cbdf075-1ce2-482f-afed-d9196a853e2f"
Jul 17 21:49:27.745071 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:27.745037    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j"]
Jul 17 21:49:27.887126 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:27.887099    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j"
Jul 17 21:49:27.990121 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:27.990083    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbdf075-1ce2-482f-afed-d9196a853e2f-maas-api-service-ca\") pod \"5cbdf075-1ce2-482f-afed-d9196a853e2f\" (UID: \"5cbdf075-1ce2-482f-afed-d9196a853e2f\") "
Jul 17 21:49:27.990545 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:27.990520    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cbdf075-1ce2-482f-afed-d9196a853e2f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5cbdf075-1ce2-482f-afed-d9196a853e2f" (UID: "5cbdf075-1ce2-482f-afed-d9196a853e2f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:49:28.091271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:28.091238    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbdf075-1ce2-482f-afed-d9196a853e2f-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:49:28.734787 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:28.734755    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j"
Jul 17 21:49:28.734974 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:28.734755    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j" event={"ID":"5cbdf075-1ce2-482f-afed-d9196a853e2f","Type":"ContainerDied","Data":"1580e9368ea3e905d990b262ef621ceb3dd2576d6cc9bc5da5087936ab38c708"}
Jul 17 21:49:28.734974 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:28.734892    2569 scope.go:117] "RemoveContainer" containerID="2c3ba40ae8be3dfff107ec40e5df9041b7c4b18a8242a1d9f95b028087bebdb5"
Jul 17 21:49:28.749580 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:28.749551    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j"]
Jul 17 21:49:28.753343 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:28.753318    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8jg6j"]
Jul 17 21:49:30.656762 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:30.656724    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cbdf075-1ce2-482f-afed-d9196a853e2f" path="/var/lib/kubelet/pods/5cbdf075-1ce2-482f-afed-d9196a853e2f/volumes"
Jul 17 21:49:39.212067 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.212024    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7"]
Jul 17 21:49:39.212520 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.212442    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cbdf075-1ce2-482f-afed-d9196a853e2f" containerName="revoke-keys"
Jul 17 21:49:39.212520 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.212456    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cbdf075-1ce2-482f-afed-d9196a853e2f" containerName="revoke-keys"
Jul 17 21:49:39.212520 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.212476    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cbdf075-1ce2-482f-afed-d9196a853e2f" containerName="revoke-keys"
Jul 17 21:49:39.212520 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.212482    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cbdf075-1ce2-482f-afed-d9196a853e2f" containerName="revoke-keys"
Jul 17 21:49:39.212520 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.212495    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3cd346b-d8b0-4dff-9f90-3975660a0a3c" containerName="revoke-keys"
Jul 17 21:49:39.212520 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.212501    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3cd346b-d8b0-4dff-9f90-3975660a0a3c" containerName="revoke-keys"
Jul 17 21:49:39.212717 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.212555    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc286d91-467e-4d8f-aa7f-6b41b51a0bc5" containerName="revoke-keys"
Jul 17 21:49:39.212717 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.212565    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cbdf075-1ce2-482f-afed-d9196a853e2f" containerName="revoke-keys"
Jul 17 21:49:39.212717 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.212571    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cbdf075-1ce2-482f-afed-d9196a853e2f" containerName="revoke-keys"
Jul 17 21:49:39.212717 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.212578    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cbdf075-1ce2-482f-afed-d9196a853e2f" containerName="revoke-keys"
Jul 17 21:49:39.216946 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.216925    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7"
Jul 17 21:49:39.219198 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.219174    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:49:39.219382 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.219352    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:49:39.222129 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.222105    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7"]
Jul 17 21:49:39.285252 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.285207    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fef25f4-d9e6-4c77-8686-dca82e5a0da6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7\" (UID: \"1fef25f4-d9e6-4c77-8686-dca82e5a0da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7"
Jul 17 21:49:39.386353 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.386315    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fef25f4-d9e6-4c77-8686-dca82e5a0da6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7\" (UID: \"1fef25f4-d9e6-4c77-8686-dca82e5a0da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7"
Jul 17 21:49:39.386963 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.386943    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fef25f4-d9e6-4c77-8686-dca82e5a0da6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7\" (UID: \"1fef25f4-d9e6-4c77-8686-dca82e5a0da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7"
Jul 17 21:49:39.528673 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.528584    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7"
Jul 17 21:49:39.664949 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.664923    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7"]
Jul 17 21:49:39.667635 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:49:39.667603    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fef25f4_d9e6_4c77_8686_dca82e5a0da6.slice/crio-e23525ee4a186548ddcc0a614f10211975e5e791048c7a0ef55ae4b7d0be3c45 WatchSource:0}: Error finding container e23525ee4a186548ddcc0a614f10211975e5e791048c7a0ef55ae4b7d0be3c45: Status 404 returned error can't find the container with id e23525ee4a186548ddcc0a614f10211975e5e791048c7a0ef55ae4b7d0be3c45
Jul 17 21:49:39.803537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:39.803451    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7" event={"ID":"1fef25f4-d9e6-4c77-8686-dca82e5a0da6","Type":"ContainerStarted","Data":"e23525ee4a186548ddcc0a614f10211975e5e791048c7a0ef55ae4b7d0be3c45"}
Jul 17 21:49:40.811636 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:40.811603    2569 generic.go:356] "Generic (PLEG): container finished" podID="1fef25f4-d9e6-4c77-8686-dca82e5a0da6" containerID="f89495879f9a3f9b722f20093971b3d3763094d1b5b9185ad9509bd79bcd0050" exitCode=6
Jul 17 21:49:40.812022 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:40.811680    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7" event={"ID":"1fef25f4-d9e6-4c77-8686-dca82e5a0da6","Type":"ContainerDied","Data":"f89495879f9a3f9b722f20093971b3d3763094d1b5b9185ad9509bd79bcd0050"}
Jul 17 21:49:40.812022 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:40.811888    2569 scope.go:117] "RemoveContainer" containerID="f89495879f9a3f9b722f20093971b3d3763094d1b5b9185ad9509bd79bcd0050"
Jul 17 21:49:41.819742 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:41.819647    2569 generic.go:356] "Generic (PLEG): container finished" podID="1fef25f4-d9e6-4c77-8686-dca82e5a0da6" containerID="bebf67284fa57556e16fcc53f5c8d60b2f13003dbf2711237a0340ecb156c06c" exitCode=6
Jul 17 21:49:41.820155 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:41.819736    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7" event={"ID":"1fef25f4-d9e6-4c77-8686-dca82e5a0da6","Type":"ContainerDied","Data":"bebf67284fa57556e16fcc53f5c8d60b2f13003dbf2711237a0340ecb156c06c"}
Jul 17 21:49:41.820155 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:41.819798    2569 scope.go:117] "RemoveContainer" containerID="f89495879f9a3f9b722f20093971b3d3763094d1b5b9185ad9509bd79bcd0050"
Jul 17 21:49:41.820155 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:41.819957    2569 scope.go:117] "RemoveContainer" containerID="bebf67284fa57556e16fcc53f5c8d60b2f13003dbf2711237a0340ecb156c06c"
Jul 17 21:49:41.820337 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:49:41.820187    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7_odh-ai-gateway-infra(1fef25f4-d9e6-4c77-8686-dca82e5a0da6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7" podUID="1fef25f4-d9e6-4c77-8686-dca82e5a0da6"
Jul 17 21:49:42.826789 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:42.826763    2569 scope.go:117] "RemoveContainer" containerID="bebf67284fa57556e16fcc53f5c8d60b2f13003dbf2711237a0340ecb156c06c"
Jul 17 21:49:42.827174 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:49:42.826963    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7_odh-ai-gateway-infra(1fef25f4-d9e6-4c77-8686-dca82e5a0da6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7" podUID="1fef25f4-d9e6-4c77-8686-dca82e5a0da6"
Jul 17 21:49:53.650378 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:53.650345    2569 scope.go:117] "RemoveContainer" containerID="bebf67284fa57556e16fcc53f5c8d60b2f13003dbf2711237a0340ecb156c06c"
Jul 17 21:49:54.896314 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:54.896278    2569 generic.go:356] "Generic (PLEG): container finished" podID="1fef25f4-d9e6-4c77-8686-dca82e5a0da6" containerID="730eecaeed61336c8ae1692fba5bd8607a9bf9103a4ea2680eb3973b8c8d2b17" exitCode=6
Jul 17 21:49:54.896813 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:54.896352    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7" event={"ID":"1fef25f4-d9e6-4c77-8686-dca82e5a0da6","Type":"ContainerDied","Data":"730eecaeed61336c8ae1692fba5bd8607a9bf9103a4ea2680eb3973b8c8d2b17"}
Jul 17 21:49:54.896813 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:54.896440    2569 scope.go:117] "RemoveContainer" containerID="bebf67284fa57556e16fcc53f5c8d60b2f13003dbf2711237a0340ecb156c06c"
Jul 17 21:49:54.896813 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:54.896791    2569 scope.go:117] "RemoveContainer" containerID="730eecaeed61336c8ae1692fba5bd8607a9bf9103a4ea2680eb3973b8c8d2b17"
Jul 17 21:49:54.897076 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:49:54.897056    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7_odh-ai-gateway-infra(1fef25f4-d9e6-4c77-8686-dca82e5a0da6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7" podUID="1fef25f4-d9e6-4c77-8686-dca82e5a0da6"
Jul 17 21:49:55.921844 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:55.921805    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7"]
Jul 17 21:49:56.062994 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:56.062971    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7"
Jul 17 21:49:56.110548 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:56.110517    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fef25f4-d9e6-4c77-8686-dca82e5a0da6-maas-api-service-ca\") pod \"1fef25f4-d9e6-4c77-8686-dca82e5a0da6\" (UID: \"1fef25f4-d9e6-4c77-8686-dca82e5a0da6\") "
Jul 17 21:49:56.110882 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:56.110861    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fef25f4-d9e6-4c77-8686-dca82e5a0da6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1fef25f4-d9e6-4c77-8686-dca82e5a0da6" (UID: "1fef25f4-d9e6-4c77-8686-dca82e5a0da6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:49:56.212159 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:56.212057    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fef25f4-d9e6-4c77-8686-dca82e5a0da6-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:49:56.911705 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:56.911656    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7" event={"ID":"1fef25f4-d9e6-4c77-8686-dca82e5a0da6","Type":"ContainerDied","Data":"e23525ee4a186548ddcc0a614f10211975e5e791048c7a0ef55ae4b7d0be3c45"}
Jul 17 21:49:56.911705 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:56.911687    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7"
Jul 17 21:49:56.911942 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:56.911738    2569 scope.go:117] "RemoveContainer" containerID="730eecaeed61336c8ae1692fba5bd8607a9bf9103a4ea2680eb3973b8c8d2b17"
Jul 17 21:49:56.926641 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:56.926615    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7"]
Jul 17 21:49:56.932057 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:56.932029    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-d6sx7"]
Jul 17 21:49:58.656829 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:49:58.656796    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fef25f4-d9e6-4c77-8686-dca82e5a0da6" path="/var/lib/kubelet/pods/1fef25f4-d9e6-4c77-8686-dca82e5a0da6/volumes"
Jul 17 21:50:04.796324 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:04.796219    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:04.894425 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:04.894338    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:07.825815 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:07.825780    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:50:10.018097 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:10.018064    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:10.695906 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:10.695871    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:16.285803 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:16.285767    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:16.351204 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:16.351155    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:16.771285 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:16.771251    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:16.875376 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:16.875342    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:16.972342 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:16.972282    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:18.465146 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:18.465116    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:21.486007 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:21.485972    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:24.416126 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:24.416089    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:24.518596 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:24.518557    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:24.616464 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:24.616433    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:24.716852 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:24.716762    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:25.094125 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:25.094094    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:29.256013 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.255969    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4"]
Jul 17 21:50:29.256496 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.256466    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cbdf075-1ce2-482f-afed-d9196a853e2f" containerName="revoke-keys"
Jul 17 21:50:29.256496 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.256481    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cbdf075-1ce2-482f-afed-d9196a853e2f" containerName="revoke-keys"
Jul 17 21:50:29.256496 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.256492    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fef25f4-d9e6-4c77-8686-dca82e5a0da6" containerName="revoke-keys"
Jul 17 21:50:29.256674 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.256501    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fef25f4-d9e6-4c77-8686-dca82e5a0da6" containerName="revoke-keys"
Jul 17 21:50:29.256674 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.256510    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fef25f4-d9e6-4c77-8686-dca82e5a0da6" containerName="revoke-keys"
Jul 17 21:50:29.256674 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.256517    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fef25f4-d9e6-4c77-8686-dca82e5a0da6" containerName="revoke-keys"
Jul 17 21:50:29.256674 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.256529    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fef25f4-d9e6-4c77-8686-dca82e5a0da6" containerName="revoke-keys"
Jul 17 21:50:29.256674 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.256534    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fef25f4-d9e6-4c77-8686-dca82e5a0da6" containerName="revoke-keys"
Jul 17 21:50:29.256674 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.256603    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fef25f4-d9e6-4c77-8686-dca82e5a0da6" containerName="revoke-keys"
Jul 17 21:50:29.256674 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.256611    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fef25f4-d9e6-4c77-8686-dca82e5a0da6" containerName="revoke-keys"
Jul 17 21:50:29.256674 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.256618    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fef25f4-d9e6-4c77-8686-dca82e5a0da6" containerName="revoke-keys"
Jul 17 21:50:29.261138 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.261119    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4"
Jul 17 21:50:29.263302 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.263210    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:50:29.265090 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.264896    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:50:29.266635 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.266611    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4"]
Jul 17 21:50:29.429520 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.429480    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff9359bc-30b3-48a4-a024-09a68cab10a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4\" (UID: \"ff9359bc-30b3-48a4-a024-09a68cab10a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4"
Jul 17 21:50:29.531017 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.530911    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff9359bc-30b3-48a4-a024-09a68cab10a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4\" (UID: \"ff9359bc-30b3-48a4-a024-09a68cab10a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4"
Jul 17 21:50:29.531721 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.531702    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff9359bc-30b3-48a4-a024-09a68cab10a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4\" (UID: \"ff9359bc-30b3-48a4-a024-09a68cab10a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4"
Jul 17 21:50:29.574124 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.574093    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4"
Jul 17 21:50:29.706941 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:29.706898    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4"]
Jul 17 21:50:29.708735 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:50:29.708704    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9359bc_30b3_48a4_a024_09a68cab10a8.slice/crio-5ee61a8690fcbb5a505e0b824df079c83a84e281be7aeedde4dc80d368e9b0c1 WatchSource:0}: Error finding container 5ee61a8690fcbb5a505e0b824df079c83a84e281be7aeedde4dc80d368e9b0c1: Status 404 returned error can't find the container with id 5ee61a8690fcbb5a505e0b824df079c83a84e281be7aeedde4dc80d368e9b0c1
Jul 17 21:50:30.111698 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:30.111653    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4" event={"ID":"ff9359bc-30b3-48a4-a024-09a68cab10a8","Type":"ContainerStarted","Data":"336d677957dded01f9e6fceac019c80797dd3745001c4a4a8b051d9746596d0b"}
Jul 17 21:50:30.111899 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:30.111714    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4" event={"ID":"ff9359bc-30b3-48a4-a024-09a68cab10a8","Type":"ContainerStarted","Data":"5ee61a8690fcbb5a505e0b824df079c83a84e281be7aeedde4dc80d368e9b0c1"}
Jul 17 21:50:30.128627 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:30.128532    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4" podStartSLOduration=1.128518205 podStartE2EDuration="1.128518205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:50:30.127103264 +0000 UTC m=+2543.886669959" watchObservedRunningTime="2026-07-17 21:50:30.128518205 +0000 UTC m=+2543.888084899"
Jul 17 21:50:31.118674 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:31.118638    2569 generic.go:356] "Generic (PLEG): container finished" podID="ff9359bc-30b3-48a4-a024-09a68cab10a8" containerID="336d677957dded01f9e6fceac019c80797dd3745001c4a4a8b051d9746596d0b" exitCode=6
Jul 17 21:50:31.119062 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:31.118686    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4" event={"ID":"ff9359bc-30b3-48a4-a024-09a68cab10a8","Type":"ContainerDied","Data":"336d677957dded01f9e6fceac019c80797dd3745001c4a4a8b051d9746596d0b"}
Jul 17 21:50:31.119062 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:31.118898    2569 scope.go:117] "RemoveContainer" containerID="336d677957dded01f9e6fceac019c80797dd3745001c4a4a8b051d9746596d0b"
Jul 17 21:50:32.125869 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:32.125836    2569 generic.go:356] "Generic (PLEG): container finished" podID="ff9359bc-30b3-48a4-a024-09a68cab10a8" containerID="1bef309a82c9f291eccad2c53920254a44a3f467d2413a1f749e30d0b6396e7b" exitCode=6
Jul 17 21:50:32.126333 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:32.125931    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4" event={"ID":"ff9359bc-30b3-48a4-a024-09a68cab10a8","Type":"ContainerDied","Data":"1bef309a82c9f291eccad2c53920254a44a3f467d2413a1f749e30d0b6396e7b"}
Jul 17 21:50:32.126333 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:32.126004    2569 scope.go:117] "RemoveContainer" containerID="336d677957dded01f9e6fceac019c80797dd3745001c4a4a8b051d9746596d0b"
Jul 17 21:50:32.126333 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:32.126109    2569 scope.go:117] "RemoveContainer" containerID="1bef309a82c9f291eccad2c53920254a44a3f467d2413a1f749e30d0b6396e7b"
Jul 17 21:50:32.126333 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:50:32.126329    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4_odh-ai-gateway-infra(ff9359bc-30b3-48a4-a024-09a68cab10a8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4" podUID="ff9359bc-30b3-48a4-a024-09a68cab10a8"
Jul 17 21:50:33.133227 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:33.133199    2569 scope.go:117] "RemoveContainer" containerID="1bef309a82c9f291eccad2c53920254a44a3f467d2413a1f749e30d0b6396e7b"
Jul 17 21:50:33.133624 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:50:33.133438    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4_odh-ai-gateway-infra(ff9359bc-30b3-48a4-a024-09a68cab10a8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4" podUID="ff9359bc-30b3-48a4-a024-09a68cab10a8"
Jul 17 21:50:47.650636 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:47.650605    2569 scope.go:117] "RemoveContainer" containerID="1bef309a82c9f291eccad2c53920254a44a3f467d2413a1f749e30d0b6396e7b"
Jul 17 21:50:48.223908 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:48.223877    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4" event={"ID":"ff9359bc-30b3-48a4-a024-09a68cab10a8","Type":"ContainerStarted","Data":"2bed46272a1f94748b566307ccb004da970f4a5827b8a175360d2b7a4b8a3a0a"}
Jul 17 21:50:49.231449 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:49.231413    2569 generic.go:356] "Generic (PLEG): container finished" podID="ff9359bc-30b3-48a4-a024-09a68cab10a8" containerID="2bed46272a1f94748b566307ccb004da970f4a5827b8a175360d2b7a4b8a3a0a" exitCode=6
Jul 17 21:50:49.231873 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:49.231473    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4" event={"ID":"ff9359bc-30b3-48a4-a024-09a68cab10a8","Type":"ContainerDied","Data":"2bed46272a1f94748b566307ccb004da970f4a5827b8a175360d2b7a4b8a3a0a"}
Jul 17 21:50:49.231873 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:49.231536    2569 scope.go:117] "RemoveContainer" containerID="1bef309a82c9f291eccad2c53920254a44a3f467d2413a1f749e30d0b6396e7b"
Jul 17 21:50:49.231873 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:49.231838    2569 scope.go:117] "RemoveContainer" containerID="2bed46272a1f94748b566307ccb004da970f4a5827b8a175360d2b7a4b8a3a0a"
Jul 17 21:50:49.232090 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:50:49.232066    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4_odh-ai-gateway-infra(ff9359bc-30b3-48a4-a024-09a68cab10a8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4" podUID="ff9359bc-30b3-48a4-a024-09a68cab10a8"
Jul 17 21:50:49.247619 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:49.247591    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4"]
Jul 17 21:50:50.382726 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:50.382696    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4"
Jul 17 21:50:50.392312 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:50.392285    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff9359bc-30b3-48a4-a024-09a68cab10a8-maas-api-service-ca\") pod \"ff9359bc-30b3-48a4-a024-09a68cab10a8\" (UID: \"ff9359bc-30b3-48a4-a024-09a68cab10a8\") "
Jul 17 21:50:50.392673 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:50.392648    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9359bc-30b3-48a4-a024-09a68cab10a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ff9359bc-30b3-48a4-a024-09a68cab10a8" (UID: "ff9359bc-30b3-48a4-a024-09a68cab10a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:50:50.493311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:50.493233    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff9359bc-30b3-48a4-a024-09a68cab10a8-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:50:51.015558 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:51.015524    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:51.246239 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:51.246207    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4"
Jul 17 21:50:51.246455 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:51.246206    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4" event={"ID":"ff9359bc-30b3-48a4-a024-09a68cab10a8","Type":"ContainerDied","Data":"5ee61a8690fcbb5a505e0b824df079c83a84e281be7aeedde4dc80d368e9b0c1"}
Jul 17 21:50:51.246455 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:51.246356    2569 scope.go:117] "RemoveContainer" containerID="2bed46272a1f94748b566307ccb004da970f4a5827b8a175360d2b7a4b8a3a0a"
Jul 17 21:50:51.262147 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:51.262113    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4"]
Jul 17 21:50:51.271689 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:51.271604    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-fjfs4"]
Jul 17 21:50:52.656452 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:52.656390    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff9359bc-30b3-48a4-a024-09a68cab10a8" path="/var/lib/kubelet/pods/ff9359bc-30b3-48a4-a024-09a68cab10a8/volumes"
Jul 17 21:50:56.743872 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:56.743841    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:56.806199 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:56.806165    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:56.895473 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:56.895434    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:56.995455 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:50:56.995337    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:07.439232 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:07.439200    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5"]
Jul 17 21:51:07.439663 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:07.439593    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9359bc-30b3-48a4-a024-09a68cab10a8" containerName="revoke-keys"
Jul 17 21:51:07.439663 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:07.439606    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9359bc-30b3-48a4-a024-09a68cab10a8" containerName="revoke-keys"
Jul 17 21:51:07.439663 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:07.439616    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9359bc-30b3-48a4-a024-09a68cab10a8" containerName="revoke-keys"
Jul 17 21:51:07.439663 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:07.439622    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9359bc-30b3-48a4-a024-09a68cab10a8" containerName="revoke-keys"
Jul 17 21:51:07.439828 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:07.439704    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff9359bc-30b3-48a4-a024-09a68cab10a8" containerName="revoke-keys"
Jul 17 21:51:07.439828 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:07.439713    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff9359bc-30b3-48a4-a024-09a68cab10a8" containerName="revoke-keys"
Jul 17 21:51:07.443853 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:07.443838    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5"
Jul 17 21:51:07.446154 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:07.446130    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:51:07.446293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:07.446186    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:51:07.450880 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:07.450853    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5"]
Jul 17 21:51:07.539021 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:07.538989    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/413d9537-74e7-4db2-9587-12778d2674d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5\" (UID: \"413d9537-74e7-4db2-9587-12778d2674d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5"
Jul 17 21:51:07.640442 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:07.640379    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/413d9537-74e7-4db2-9587-12778d2674d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5\" (UID: \"413d9537-74e7-4db2-9587-12778d2674d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5"
Jul 17 21:51:07.641063 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:07.641035    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/413d9537-74e7-4db2-9587-12778d2674d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5\" (UID: \"413d9537-74e7-4db2-9587-12778d2674d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5"
Jul 17 21:51:07.756369 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:07.756283    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5"
Jul 17 21:51:07.896154 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:07.895692    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5"]
Jul 17 21:51:08.340661 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:08.340623    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5" event={"ID":"413d9537-74e7-4db2-9587-12778d2674d8","Type":"ContainerStarted","Data":"2ce7db62d2e12e165dcde2d5d9961bcd3fe6fb8903c6b8c7839148dd7425b141"}
Jul 17 21:51:08.340836 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:08.340687    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5" event={"ID":"413d9537-74e7-4db2-9587-12778d2674d8","Type":"ContainerStarted","Data":"1bf43f3d0293d19d250cec37c899984e017456767a7410161f23ff259b69acce"}
Jul 17 21:51:08.359879 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:08.359825    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5" podStartSLOduration=1.359808471 podStartE2EDuration="1.359808471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:51:08.35633346 +0000 UTC m=+2582.115900154" watchObservedRunningTime="2026-07-17 21:51:08.359808471 +0000 UTC m=+2582.119375164"
Jul 17 21:51:09.348393 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:09.348356    2569 generic.go:356] "Generic (PLEG): container finished" podID="413d9537-74e7-4db2-9587-12778d2674d8" containerID="2ce7db62d2e12e165dcde2d5d9961bcd3fe6fb8903c6b8c7839148dd7425b141" exitCode=6
Jul 17 21:51:09.348857 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:09.348436    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5" event={"ID":"413d9537-74e7-4db2-9587-12778d2674d8","Type":"ContainerDied","Data":"2ce7db62d2e12e165dcde2d5d9961bcd3fe6fb8903c6b8c7839148dd7425b141"}
Jul 17 21:51:09.348857 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:09.348664    2569 scope.go:117] "RemoveContainer" containerID="2ce7db62d2e12e165dcde2d5d9961bcd3fe6fb8903c6b8c7839148dd7425b141"
Jul 17 21:51:10.355875 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:10.355843    2569 generic.go:356] "Generic (PLEG): container finished" podID="413d9537-74e7-4db2-9587-12778d2674d8" containerID="8aa22c7e525096a39030760fef175943f104491b4d15f171aa7e0b10e503a80e" exitCode=6
Jul 17 21:51:10.356257 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:10.355933    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5" event={"ID":"413d9537-74e7-4db2-9587-12778d2674d8","Type":"ContainerDied","Data":"8aa22c7e525096a39030760fef175943f104491b4d15f171aa7e0b10e503a80e"}
Jul 17 21:51:10.356257 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:10.355997    2569 scope.go:117] "RemoveContainer" containerID="2ce7db62d2e12e165dcde2d5d9961bcd3fe6fb8903c6b8c7839148dd7425b141"
Jul 17 21:51:10.356257 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:10.356164    2569 scope.go:117] "RemoveContainer" containerID="8aa22c7e525096a39030760fef175943f104491b4d15f171aa7e0b10e503a80e"
Jul 17 21:51:10.356428 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:51:10.356390    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5_odh-ai-gateway-infra(413d9537-74e7-4db2-9587-12778d2674d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5" podUID="413d9537-74e7-4db2-9587-12778d2674d8"
Jul 17 21:51:11.364489 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:11.364457    2569 scope.go:117] "RemoveContainer" containerID="8aa22c7e525096a39030760fef175943f104491b4d15f171aa7e0b10e503a80e"
Jul 17 21:51:11.364886 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:51:11.364641    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5_odh-ai-gateway-infra(413d9537-74e7-4db2-9587-12778d2674d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5" podUID="413d9537-74e7-4db2-9587-12778d2674d8"
Jul 17 21:51:12.841697 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:12.841658    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:18.815109 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:18.815075    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:18.919961 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:18.919918    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:19.015691 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:19.015653    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:26.653567 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:26.653533    2569 scope.go:117] "RemoveContainer" containerID="8aa22c7e525096a39030760fef175943f104491b4d15f171aa7e0b10e503a80e"
Jul 17 21:51:27.455300 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:27.455266    2569 generic.go:356] "Generic (PLEG): container finished" podID="413d9537-74e7-4db2-9587-12778d2674d8" containerID="3175e21306de1ade0f19469a6e2c43c247fac1389e343692c85244359adc2e1a" exitCode=6
Jul 17 21:51:27.455523 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:27.455346    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5" event={"ID":"413d9537-74e7-4db2-9587-12778d2674d8","Type":"ContainerDied","Data":"3175e21306de1ade0f19469a6e2c43c247fac1389e343692c85244359adc2e1a"}
Jul 17 21:51:27.455523 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:27.455433    2569 scope.go:117] "RemoveContainer" containerID="8aa22c7e525096a39030760fef175943f104491b4d15f171aa7e0b10e503a80e"
Jul 17 21:51:27.455749 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:27.455734    2569 scope.go:117] "RemoveContainer" containerID="3175e21306de1ade0f19469a6e2c43c247fac1389e343692c85244359adc2e1a"
Jul 17 21:51:27.455963 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:51:27.455943    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5_odh-ai-gateway-infra(413d9537-74e7-4db2-9587-12778d2674d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5" podUID="413d9537-74e7-4db2-9587-12778d2674d8"
Jul 17 21:51:28.483333 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:28.483295    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5"]
Jul 17 21:51:28.628265 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:28.628237    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5"
Jul 17 21:51:28.729452 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:28.729380    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/413d9537-74e7-4db2-9587-12778d2674d8-maas-api-service-ca\") pod \"413d9537-74e7-4db2-9587-12778d2674d8\" (UID: \"413d9537-74e7-4db2-9587-12778d2674d8\") "
Jul 17 21:51:28.729753 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:28.729730    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/413d9537-74e7-4db2-9587-12778d2674d8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "413d9537-74e7-4db2-9587-12778d2674d8" (UID: "413d9537-74e7-4db2-9587-12778d2674d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:51:28.830093 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:28.830064    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/413d9537-74e7-4db2-9587-12778d2674d8-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:51:29.469081 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:29.469046    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5"
Jul 17 21:51:29.469274 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:29.469047    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5" event={"ID":"413d9537-74e7-4db2-9587-12778d2674d8","Type":"ContainerDied","Data":"1bf43f3d0293d19d250cec37c899984e017456767a7410161f23ff259b69acce"}
Jul 17 21:51:29.469274 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:29.469192    2569 scope.go:117] "RemoveContainer" containerID="3175e21306de1ade0f19469a6e2c43c247fac1389e343692c85244359adc2e1a"
Jul 17 21:51:29.490754 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:29.490728    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5"]
Jul 17 21:51:29.499484 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:29.499452    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-bgxk5"]
Jul 17 21:51:29.519860 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:29.519834    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:30.294594 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:30.294561    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:30.656753 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:30.656712    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="413d9537-74e7-4db2-9587-12778d2674d8" path="/var/lib/kubelet/pods/413d9537-74e7-4db2-9587-12778d2674d8/volumes"
Jul 17 21:51:36.161316 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:36.161282    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:36.265764 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:36.265733    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:36.361938 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:36.361908    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:36.462084 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:36.462002    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:37.203238 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:37.203198    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:37.299755 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:37.299720    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:37.399896 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:37.399858    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:38.321831 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:38.321802    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:51:38.999591 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:38.999560    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:44.705114 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:44.705080    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:44.803748 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:44.803713    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:45.363292 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:45.363254    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:47.616006 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:47.615966    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:51:50.528448 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.528410    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s"]
Jul 17 21:51:50.528962 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.528809    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="413d9537-74e7-4db2-9587-12778d2674d8" containerName="revoke-keys"
Jul 17 21:51:50.528962 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.528820    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="413d9537-74e7-4db2-9587-12778d2674d8" containerName="revoke-keys"
Jul 17 21:51:50.528962 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.528828    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="413d9537-74e7-4db2-9587-12778d2674d8" containerName="revoke-keys"
Jul 17 21:51:50.528962 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.528834    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="413d9537-74e7-4db2-9587-12778d2674d8" containerName="revoke-keys"
Jul 17 21:51:50.528962 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.528843    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="413d9537-74e7-4db2-9587-12778d2674d8" containerName="revoke-keys"
Jul 17 21:51:50.528962 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.528849    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="413d9537-74e7-4db2-9587-12778d2674d8" containerName="revoke-keys"
Jul 17 21:51:50.528962 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.528869    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9359bc-30b3-48a4-a024-09a68cab10a8" containerName="revoke-keys"
Jul 17 21:51:50.528962 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.528874    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9359bc-30b3-48a4-a024-09a68cab10a8" containerName="revoke-keys"
Jul 17 21:51:50.528962 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.528933    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="413d9537-74e7-4db2-9587-12778d2674d8" containerName="revoke-keys"
Jul 17 21:51:50.528962 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.528941    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="413d9537-74e7-4db2-9587-12778d2674d8" containerName="revoke-keys"
Jul 17 21:51:50.528962 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.528947    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff9359bc-30b3-48a4-a024-09a68cab10a8" containerName="revoke-keys"
Jul 17 21:51:50.532009 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.531991    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s"
Jul 17 21:51:50.534134 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.534110    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:51:50.534134 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.534132    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:51:50.540700 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.540680    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s"]
Jul 17 21:51:50.632934 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.632895    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf8d03a-78c3-4a54-a30e-42829ad6624e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-sr79s\" (UID: \"7bf8d03a-78c3-4a54-a30e-42829ad6624e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s"
Jul 17 21:51:50.734315 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.734273    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf8d03a-78c3-4a54-a30e-42829ad6624e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-sr79s\" (UID: \"7bf8d03a-78c3-4a54-a30e-42829ad6624e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s"
Jul 17 21:51:50.735006 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.734983    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf8d03a-78c3-4a54-a30e-42829ad6624e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-sr79s\" (UID: \"7bf8d03a-78c3-4a54-a30e-42829ad6624e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s"
Jul 17 21:51:50.843665 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.843633    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s"
Jul 17 21:51:50.981279 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:50.981252    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s"]
Jul 17 21:51:50.983258 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:51:50.983230    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf8d03a_78c3_4a54_a30e_42829ad6624e.slice/crio-cb4fb248f329314ed551214d77d04da75b22b741a03c67c4c42a5b51cf114d8f WatchSource:0}: Error finding container cb4fb248f329314ed551214d77d04da75b22b741a03c67c4c42a5b51cf114d8f: Status 404 returned error can't find the container with id cb4fb248f329314ed551214d77d04da75b22b741a03c67c4c42a5b51cf114d8f
Jul 17 21:51:51.457246 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:51.457115    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:51.605073 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:51.605039    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s" event={"ID":"7bf8d03a-78c3-4a54-a30e-42829ad6624e","Type":"ContainerStarted","Data":"fdfbba6a01d51241bddb57a3e64cb472595df2bc539329a09f8ca8868562888c"}
Jul 17 21:51:51.605556 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:51.605122    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s" event={"ID":"7bf8d03a-78c3-4a54-a30e-42829ad6624e","Type":"ContainerStarted","Data":"cb4fb248f329314ed551214d77d04da75b22b741a03c67c4c42a5b51cf114d8f"}
Jul 17 21:51:51.632539 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:51.632488    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s" podStartSLOduration=1.632471062 podStartE2EDuration="1.632471062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:51:51.63078781 +0000 UTC m=+2625.390354529" watchObservedRunningTime="2026-07-17 21:51:51.632471062 +0000 UTC m=+2625.392037757"
Jul 17 21:51:52.588952 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:52.588911    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:52.612899 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:52.612859    2569 generic.go:356] "Generic (PLEG): container finished" podID="7bf8d03a-78c3-4a54-a30e-42829ad6624e" containerID="fdfbba6a01d51241bddb57a3e64cb472595df2bc539329a09f8ca8868562888c" exitCode=6
Jul 17 21:51:52.613308 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:52.612943    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s" event={"ID":"7bf8d03a-78c3-4a54-a30e-42829ad6624e","Type":"ContainerDied","Data":"fdfbba6a01d51241bddb57a3e64cb472595df2bc539329a09f8ca8868562888c"}
Jul 17 21:51:52.613308 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:52.613185    2569 scope.go:117] "RemoveContainer" containerID="fdfbba6a01d51241bddb57a3e64cb472595df2bc539329a09f8ca8868562888c"
Jul 17 21:51:53.096171 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:53.096138    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:53.620962 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:53.620926    2569 generic.go:356] "Generic (PLEG): container finished" podID="7bf8d03a-78c3-4a54-a30e-42829ad6624e" containerID="25ed1a63c2ac3c1bb815cccd8a967713f104d0db9c7be6ca17a034a91b87982c" exitCode=6
Jul 17 21:51:53.621415 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:53.620987    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s" event={"ID":"7bf8d03a-78c3-4a54-a30e-42829ad6624e","Type":"ContainerDied","Data":"25ed1a63c2ac3c1bb815cccd8a967713f104d0db9c7be6ca17a034a91b87982c"}
Jul 17 21:51:53.621415 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:53.621029    2569 scope.go:117] "RemoveContainer" containerID="fdfbba6a01d51241bddb57a3e64cb472595df2bc539329a09f8ca8868562888c"
Jul 17 21:51:53.621415 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:53.621176    2569 scope.go:117] "RemoveContainer" containerID="25ed1a63c2ac3c1bb815cccd8a967713f104d0db9c7be6ca17a034a91b87982c"
Jul 17 21:51:53.621549 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:51:53.621453    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-sr79s_odh-ai-gateway-infra(7bf8d03a-78c3-4a54-a30e-42829ad6624e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s" podUID="7bf8d03a-78c3-4a54-a30e-42829ad6624e"
Jul 17 21:51:54.629111 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:54.629085    2569 scope.go:117] "RemoveContainer" containerID="25ed1a63c2ac3c1bb815cccd8a967713f104d0db9c7be6ca17a034a91b87982c"
Jul 17 21:51:54.629514 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:51:54.629302    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-sr79s_odh-ai-gateway-infra(7bf8d03a-78c3-4a54-a30e-42829ad6624e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s" podUID="7bf8d03a-78c3-4a54-a30e-42829ad6624e"
Jul 17 21:51:54.769626 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:54.769590    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:54.859098 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:54.859057    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:55.062870 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:55.062780    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:55.180744 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:55.180709    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:55.556963 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:55.556932    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:55.656856 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:55.656828    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:59.217248 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:59.217212    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:59.819147 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:59.819109    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:51:59.874649 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:59.874602    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:59.970685 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:51:59.970649    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:00.270381 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:00.270344    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:05.905153 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:05.905119    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:05.965027 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:05.964997    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:06.257110 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:06.257025    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62f6c7d6-mxzcp"]
Jul 17 21:52:06.257519 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:06.257504    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="413d9537-74e7-4db2-9587-12778d2674d8" containerName="revoke-keys"
Jul 17 21:52:06.262050 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:06.262021    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62f6c7d6-mxzcp"
Jul 17 21:52:06.268062 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:06.268030    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62f6c7d6-mxzcp"]
Jul 17 21:52:06.389194 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:06.389154    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d39f53-5de5-4702-baaa-a02b744153c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-62f6c7d6-mxzcp\" (UID: \"d7d39f53-5de5-4702-baaa-a02b744153c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62f6c7d6-mxzcp"
Jul 17 21:52:06.490372 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:06.490328    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d39f53-5de5-4702-baaa-a02b744153c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-62f6c7d6-mxzcp\" (UID: \"d7d39f53-5de5-4702-baaa-a02b744153c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62f6c7d6-mxzcp"
Jul 17 21:52:06.491049 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:06.491025    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d39f53-5de5-4702-baaa-a02b744153c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-62f6c7d6-mxzcp\" (UID: \"d7d39f53-5de5-4702-baaa-a02b744153c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62f6c7d6-mxzcp"
Jul 17 21:52:06.576369 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:06.576280    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62f6c7d6-mxzcp"
Jul 17 21:52:06.714565 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:06.714542    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62f6c7d6-mxzcp"]
Jul 17 21:52:06.716593 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:52:06.716569    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d39f53_5de5_4702_baaa_a02b744153c1.slice/crio-17bbcad1dafce1b875f48eccfa4706bfb3c8708ec774415becb14025d90aed5d WatchSource:0}: Error finding container 17bbcad1dafce1b875f48eccfa4706bfb3c8708ec774415becb14025d90aed5d: Status 404 returned error can't find the container with id 17bbcad1dafce1b875f48eccfa4706bfb3c8708ec774415becb14025d90aed5d
Jul 17 21:52:07.650937 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:07.650851    2569 scope.go:117] "RemoveContainer" containerID="25ed1a63c2ac3c1bb815cccd8a967713f104d0db9c7be6ca17a034a91b87982c"
Jul 17 21:52:07.710231 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:07.710195    2569 generic.go:356] "Generic (PLEG): container finished" podID="d7d39f53-5de5-4702-baaa-a02b744153c1" containerID="02bf1bf5e22b400cd78b1f942b42a8931d18045bce8a9abc870066a162583893" exitCode=0
Jul 17 21:52:07.710433 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:07.710300    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62f6c7d6-mxzcp" event={"ID":"d7d39f53-5de5-4702-baaa-a02b744153c1","Type":"ContainerDied","Data":"02bf1bf5e22b400cd78b1f942b42a8931d18045bce8a9abc870066a162583893"}
Jul 17 21:52:07.710433 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:07.710360    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62f6c7d6-mxzcp" event={"ID":"d7d39f53-5de5-4702-baaa-a02b744153c1","Type":"ContainerStarted","Data":"17bbcad1dafce1b875f48eccfa4706bfb3c8708ec774415becb14025d90aed5d"}
Jul 17 21:52:08.718818 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:08.718725    2569 generic.go:356] "Generic (PLEG): container finished" podID="7bf8d03a-78c3-4a54-a30e-42829ad6624e" containerID="6e4f9302fa44c98daef1340d448af56b260e6fd57510f61ab2ee510cc82e5c70" exitCode=6
Jul 17 21:52:08.718818 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:08.718793    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s" event={"ID":"7bf8d03a-78c3-4a54-a30e-42829ad6624e","Type":"ContainerDied","Data":"6e4f9302fa44c98daef1340d448af56b260e6fd57510f61ab2ee510cc82e5c70"}
Jul 17 21:52:08.719342 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:08.718856    2569 scope.go:117] "RemoveContainer" containerID="25ed1a63c2ac3c1bb815cccd8a967713f104d0db9c7be6ca17a034a91b87982c"
Jul 17 21:52:08.719342 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:08.719177    2569 scope.go:117] "RemoveContainer" containerID="6e4f9302fa44c98daef1340d448af56b260e6fd57510f61ab2ee510cc82e5c70"
Jul 17 21:52:08.719821 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:52:08.719432    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-sr79s_odh-ai-gateway-infra(7bf8d03a-78c3-4a54-a30e-42829ad6624e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s" podUID="7bf8d03a-78c3-4a54-a30e-42829ad6624e"
Jul 17 21:52:08.865862 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:08.865835    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62f6c7d6-mxzcp"
Jul 17 21:52:09.013839 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:09.013744    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d39f53-5de5-4702-baaa-a02b744153c1-maas-api-service-ca\") pod \"d7d39f53-5de5-4702-baaa-a02b744153c1\" (UID: \"d7d39f53-5de5-4702-baaa-a02b744153c1\") "
Jul 17 21:52:09.014146 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:09.014119    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d39f53-5de5-4702-baaa-a02b744153c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d7d39f53-5de5-4702-baaa-a02b744153c1" (UID: "d7d39f53-5de5-4702-baaa-a02b744153c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:52:09.115500 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:09.115457    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d39f53-5de5-4702-baaa-a02b744153c1-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:52:09.726309 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:09.726268    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62f6c7d6-mxzcp" event={"ID":"d7d39f53-5de5-4702-baaa-a02b744153c1","Type":"ContainerDied","Data":"17bbcad1dafce1b875f48eccfa4706bfb3c8708ec774415becb14025d90aed5d"}
Jul 17 21:52:09.726895 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:09.726873    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17bbcad1dafce1b875f48eccfa4706bfb3c8708ec774415becb14025d90aed5d"
Jul 17 21:52:09.727074 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:09.727057    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62f6c7d6-mxzcp"
Jul 17 21:52:09.745928 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:09.745887    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s"]
Jul 17 21:52:09.883250 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:09.883223    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s"
Jul 17 21:52:10.022328 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:10.022239    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf8d03a-78c3-4a54-a30e-42829ad6624e-maas-api-service-ca\") pod \"7bf8d03a-78c3-4a54-a30e-42829ad6624e\" (UID: \"7bf8d03a-78c3-4a54-a30e-42829ad6624e\") "
Jul 17 21:52:10.022656 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:10.022635    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf8d03a-78c3-4a54-a30e-42829ad6624e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7bf8d03a-78c3-4a54-a30e-42829ad6624e" (UID: "7bf8d03a-78c3-4a54-a30e-42829ad6624e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:52:10.123177 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:10.123147    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf8d03a-78c3-4a54-a30e-42829ad6624e-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:52:10.739089 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:10.739058    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s"
Jul 17 21:52:10.739549 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:10.739099    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s" event={"ID":"7bf8d03a-78c3-4a54-a30e-42829ad6624e","Type":"ContainerDied","Data":"cb4fb248f329314ed551214d77d04da75b22b741a03c67c4c42a5b51cf114d8f"}
Jul 17 21:52:10.739549 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:10.739162    2569 scope.go:117] "RemoveContainer" containerID="6e4f9302fa44c98daef1340d448af56b260e6fd57510f61ab2ee510cc82e5c70"
Jul 17 21:52:10.756792 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:10.756763    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s"]
Jul 17 21:52:10.761970 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:10.761947    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-sr79s"]
Jul 17 21:52:12.656718 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:12.656687    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bf8d03a-78c3-4a54-a30e-42829ad6624e" path="/var/lib/kubelet/pods/7bf8d03a-78c3-4a54-a30e-42829ad6624e/volumes"
Jul 17 21:52:16.373155 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:16.373125    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:19.215347 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:19.215307    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:52:21.338999 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:21.338960    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:27.033238 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:27.033200    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:27.133988 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:27.133951    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:32.011198 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:32.011149    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:32.230500 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:32.230465    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:32.347760 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:32.347709    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:32.447843 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:32.447814    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:32.574079 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:32.574042    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:38.459738 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:38.459704    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:38.517577 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:38.517545    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:38.599743 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:38.599707    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:38.699833 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:38.699798    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:38.823039 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:38.822959    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:44.599003 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:44.598968    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:44.699663 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:44.699629    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:51.254425 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:51.254368    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:51.545960 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:51.545248    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:51.642270 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:51.642228    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:54.171177 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:54.171144    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:54.274840 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:54.274801    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:54.372063 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:54.372011    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:54.472145 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:54.472055    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:54.855788 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:54.855755    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:54.952740 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:54.952704    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:57.420628 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:57.420590    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:52:58.544610 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:52:58.544580    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:06.870190 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:06.870115    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:53:06.893463 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:06.893439    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:53:08.222081 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.222044    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"]
Jul 17 21:53:08.222472 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.222456    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7d39f53-5de5-4702-baaa-a02b744153c1" containerName="revoke-keys"
Jul 17 21:53:08.222472 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.222468    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7d39f53-5de5-4702-baaa-a02b744153c1" containerName="revoke-keys"
Jul 17 21:53:08.222601 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.222477    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bf8d03a-78c3-4a54-a30e-42829ad6624e" containerName="revoke-keys"
Jul 17 21:53:08.222601 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.222482    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bf8d03a-78c3-4a54-a30e-42829ad6624e" containerName="revoke-keys"
Jul 17 21:53:08.222601 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.222505    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bf8d03a-78c3-4a54-a30e-42829ad6624e" containerName="revoke-keys"
Jul 17 21:53:08.222601 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.222511    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bf8d03a-78c3-4a54-a30e-42829ad6624e" containerName="revoke-keys"
Jul 17 21:53:08.222601 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.222571    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7d39f53-5de5-4702-baaa-a02b744153c1" containerName="revoke-keys"
Jul 17 21:53:08.222601 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.222580    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bf8d03a-78c3-4a54-a30e-42829ad6624e" containerName="revoke-keys"
Jul 17 21:53:08.222601 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.222586    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bf8d03a-78c3-4a54-a30e-42829ad6624e" containerName="revoke-keys"
Jul 17 21:53:08.222601 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.222591    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bf8d03a-78c3-4a54-a30e-42829ad6624e" containerName="revoke-keys"
Jul 17 21:53:08.225999 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.225975    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:08.228790 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.228763    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c39f8651\"/\"e2e-lifecycle-model-c39f8651-kserve-self-signed-certs\""
Jul 17 21:53:08.228932 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.228828    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c39f8651\"/\"default-dockercfg-hmjk2\""
Jul 17 21:53:08.228932 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.228842    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c39f8651\"/\"kube-root-ca.crt\""
Jul 17 21:53:08.228932 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.228828    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c39f8651\"/\"openshift-service-ca.crt\""
Jul 17 21:53:08.235271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.235244    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"]
Jul 17 21:53:08.253676 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.253637    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:08.319932 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.319888    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-tls-certs\") pod \"e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") " pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:08.320119 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.319982    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-model-cache\") pod \"e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") " pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:08.320119 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.320054    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sjx7\" (UniqueName: \"kubernetes.io/projected/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-kube-api-access-6sjx7\") pod \"e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") " pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:08.320206 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.320145    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-tmp-dir\") pod \"e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") " pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:08.320206 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.320176    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-dshm\") pod \"e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") " pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:08.320292 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.320209    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-home\") pod \"e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") " pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:08.421435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.421375    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-model-cache\") pod \"e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") " pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:08.421630 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.421453    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sjx7\" (UniqueName: \"kubernetes.io/projected/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-kube-api-access-6sjx7\") pod \"e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") " pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:08.421630 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.421511    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-tmp-dir\") pod \"e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") " pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:08.421630 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.421534    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-dshm\") pod \"e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") " pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:08.421630 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.421553    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-home\") pod \"e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") " pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:08.421630 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.421575    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-tls-certs\") pod \"e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") " pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:08.421896 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.421818    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-model-cache\") pod \"e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") " pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:08.421955 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.421911    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-tmp-dir\") pod \"e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") " pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:08.421955 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.421942    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-home\") pod \"e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") " pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:08.423900 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.423874    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-dshm\") pod \"e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") " pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:08.424202 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.424183    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-tls-certs\") pod \"e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") " pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:08.429681 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.429659    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sjx7\" (UniqueName: \"kubernetes.io/projected/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-kube-api-access-6sjx7\") pod \"e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") " pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:08.537182 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.537082    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:08.679747 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:08.679706    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"]
Jul 17 21:53:08.682487 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:53:08.682458    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55ce6d82_5eb2_4d44_8a9a_44eee751cb8b.slice/crio-fab1731221049b65d962618d8db162fc4564c75b23c4abc63bebfe449452fd2f WatchSource:0}: Error finding container fab1731221049b65d962618d8db162fc4564c75b23c4abc63bebfe449452fd2f: Status 404 returned error can't find the container with id fab1731221049b65d962618d8db162fc4564c75b23c4abc63bebfe449452fd2f
Jul 17 21:53:09.077264 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:09.077224    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m" event={"ID":"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b","Type":"ContainerStarted","Data":"372e934ad43ab70431766ab520b9fdcddb49f75c7f9c73728b884ae43119149d"}
Jul 17 21:53:09.077264 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:09.077278    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m" event={"ID":"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b","Type":"ContainerStarted","Data":"fab1731221049b65d962618d8db162fc4564c75b23c4abc63bebfe449452fd2f"}
Jul 17 21:53:09.097085 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:09.097031    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m" podStartSLOduration=1.097011952 podStartE2EDuration="1.097011952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:09.093908519 +0000 UTC m=+2702.853475212" watchObservedRunningTime="2026-07-17 21:53:09.097011952 +0000 UTC m=+2702.856578646"
Jul 17 21:53:09.405546 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:09.405503    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs"]
Jul 17 21:53:09.406083 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:09.406064    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bf8d03a-78c3-4a54-a30e-42829ad6624e" containerName="revoke-keys"
Jul 17 21:53:09.406163 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:09.406088    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bf8d03a-78c3-4a54-a30e-42829ad6624e" containerName="revoke-keys"
Jul 17 21:53:09.409381 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:09.409356    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs"
Jul 17 21:53:09.411437 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:09.411331    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:53:09.411437 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:09.411367    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:53:09.412268 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:09.412243    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs"]
Jul 17 21:53:09.532666 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:09.532621    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9551a349-e1e5-497a-bf88-4c417bc6390d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs\" (UID: \"9551a349-e1e5-497a-bf88-4c417bc6390d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs"
Jul 17 21:53:09.634087 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:09.634049    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9551a349-e1e5-497a-bf88-4c417bc6390d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs\" (UID: \"9551a349-e1e5-497a-bf88-4c417bc6390d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs"
Jul 17 21:53:09.634671 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:09.634653    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9551a349-e1e5-497a-bf88-4c417bc6390d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs\" (UID: \"9551a349-e1e5-497a-bf88-4c417bc6390d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs"
Jul 17 21:53:09.722248 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:09.722171    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs"
Jul 17 21:53:09.863856 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:09.863825    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs"]
Jul 17 21:53:09.865024 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:53:09.864999    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9551a349_e1e5_497a_bf88_4c417bc6390d.slice/crio-3c38aba80f0dc02566c4132d71b254eafa28fbfa517226831133201177812f7c WatchSource:0}: Error finding container 3c38aba80f0dc02566c4132d71b254eafa28fbfa517226831133201177812f7c: Status 404 returned error can't find the container with id 3c38aba80f0dc02566c4132d71b254eafa28fbfa517226831133201177812f7c
Jul 17 21:53:10.084500 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:10.084466    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs" event={"ID":"9551a349-e1e5-497a-bf88-4c417bc6390d","Type":"ContainerStarted","Data":"3c38aba80f0dc02566c4132d71b254eafa28fbfa517226831133201177812f7c"}
Jul 17 21:53:11.091510 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:11.091470    2569 generic.go:356] "Generic (PLEG): container finished" podID="9551a349-e1e5-497a-bf88-4c417bc6390d" containerID="bccd09ed80f88aea52c77c8a18d9b6094c7d54d4248528026c2799c9a9385dc0" exitCode=6
Jul 17 21:53:11.091977 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:11.091559    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs" event={"ID":"9551a349-e1e5-497a-bf88-4c417bc6390d","Type":"ContainerDied","Data":"bccd09ed80f88aea52c77c8a18d9b6094c7d54d4248528026c2799c9a9385dc0"}
Jul 17 21:53:11.091977 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:11.091734    2569 scope.go:117] "RemoveContainer" containerID="bccd09ed80f88aea52c77c8a18d9b6094c7d54d4248528026c2799c9a9385dc0"
Jul 17 21:53:11.611589 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:11.611555    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:53:12.099044 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:12.099005    2569 generic.go:356] "Generic (PLEG): container finished" podID="9551a349-e1e5-497a-bf88-4c417bc6390d" containerID="d8a9c5c3c2e8b53d21ae692f0b7c6b031e38ad3192ad1b293304fd223b4b18a9" exitCode=6
Jul 17 21:53:12.099513 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:12.099102    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs" event={"ID":"9551a349-e1e5-497a-bf88-4c417bc6390d","Type":"ContainerDied","Data":"d8a9c5c3c2e8b53d21ae692f0b7c6b031e38ad3192ad1b293304fd223b4b18a9"}
Jul 17 21:53:12.100440 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:12.099183    2569 scope.go:117] "RemoveContainer" containerID="bccd09ed80f88aea52c77c8a18d9b6094c7d54d4248528026c2799c9a9385dc0"
Jul 17 21:53:12.100538 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:12.100522    2569 scope.go:117] "RemoveContainer" containerID="d8a9c5c3c2e8b53d21ae692f0b7c6b031e38ad3192ad1b293304fd223b4b18a9"
Jul 17 21:53:12.100778 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:53:12.100756    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs_odh-ai-gateway-infra(9551a349-e1e5-497a-bf88-4c417bc6390d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs" podUID="9551a349-e1e5-497a-bf88-4c417bc6390d"
Jul 17 21:53:13.106589 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:13.106563    2569 scope.go:117] "RemoveContainer" containerID="d8a9c5c3c2e8b53d21ae692f0b7c6b031e38ad3192ad1b293304fd223b4b18a9"
Jul 17 21:53:13.106939 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:53:13.106740    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs_odh-ai-gateway-infra(9551a349-e1e5-497a-bf88-4c417bc6390d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs" podUID="9551a349-e1e5-497a-bf88-4c417bc6390d"
Jul 17 21:53:18.538117 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:18.538075    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:18.538561 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:18.538131    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:18.550948 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:18.550921    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:19.157339 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:19.157311    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:53:24.714438 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:24.714384    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:53:25.307848 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:25.307814    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8"]
Jul 17 21:53:25.311722 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:25.311663    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8"
Jul 17 21:53:25.319270 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:25.319248    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8"]
Jul 17 21:53:25.486767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:25.486731    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba578e55-e5ca-40e2-85ac-4e090562697d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-446f8\" (UID: \"ba578e55-e5ca-40e2-85ac-4e090562697d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8"
Jul 17 21:53:25.587854 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:25.587820    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba578e55-e5ca-40e2-85ac-4e090562697d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-446f8\" (UID: \"ba578e55-e5ca-40e2-85ac-4e090562697d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8"
Jul 17 21:53:25.588664 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:25.588640    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba578e55-e5ca-40e2-85ac-4e090562697d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-446f8\" (UID: \"ba578e55-e5ca-40e2-85ac-4e090562697d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8"
Jul 17 21:53:25.625801 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:25.625759    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8"
Jul 17 21:53:25.771996 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:25.771969    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8"]
Jul 17 21:53:25.774234 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:53:25.774201    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba578e55_e5ca_40e2_85ac_4e090562697d.slice/crio-4eba6757a325b58591b7db782a6aa333612faef4348808c02563182c34214381 WatchSource:0}: Error finding container 4eba6757a325b58591b7db782a6aa333612faef4348808c02563182c34214381: Status 404 returned error can't find the container with id 4eba6757a325b58591b7db782a6aa333612faef4348808c02563182c34214381
Jul 17 21:53:26.188541 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:26.188438    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8" event={"ID":"ba578e55-e5ca-40e2-85ac-4e090562697d","Type":"ContainerStarted","Data":"b8c0c1adca11c3b3f4024c5eadab3375287c3bffbd9ebff9ce67397648cb9925"}
Jul 17 21:53:26.188541 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:26.188511    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8" event={"ID":"ba578e55-e5ca-40e2-85ac-4e090562697d","Type":"ContainerStarted","Data":"4eba6757a325b58591b7db782a6aa333612faef4348808c02563182c34214381"}
Jul 17 21:53:26.202634 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:26.202577    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8" podStartSLOduration=1.202561532 podStartE2EDuration="1.202561532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:26.200463367 +0000 UTC m=+2719.960030061" watchObservedRunningTime="2026-07-17 21:53:26.202561532 +0000 UTC m=+2719.962128266"
Jul 17 21:53:26.655440 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:26.655387    2569 scope.go:117] "RemoveContainer" containerID="d8a9c5c3c2e8b53d21ae692f0b7c6b031e38ad3192ad1b293304fd223b4b18a9"
Jul 17 21:53:27.196444 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:27.196392    2569 generic.go:356] "Generic (PLEG): container finished" podID="ba578e55-e5ca-40e2-85ac-4e090562697d" containerID="b8c0c1adca11c3b3f4024c5eadab3375287c3bffbd9ebff9ce67397648cb9925" exitCode=6
Jul 17 21:53:27.196948 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:27.196493    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8" event={"ID":"ba578e55-e5ca-40e2-85ac-4e090562697d","Type":"ContainerDied","Data":"b8c0c1adca11c3b3f4024c5eadab3375287c3bffbd9ebff9ce67397648cb9925"}
Jul 17 21:53:27.196948 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:27.196718    2569 scope.go:117] "RemoveContainer" containerID="b8c0c1adca11c3b3f4024c5eadab3375287c3bffbd9ebff9ce67397648cb9925"
Jul 17 21:53:27.198492 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:27.198468    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs" event={"ID":"9551a349-e1e5-497a-bf88-4c417bc6390d","Type":"ContainerStarted","Data":"e93b254a1e6de21b02a6199770d90d85de48748cf95e0247a535fe407888c87d"}
Jul 17 21:53:27.223997 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:27.223945    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs" podStartSLOduration=18.22392921 podStartE2EDuration="18.22392921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:27.222305434 +0000 UTC m=+2720.981872127" watchObservedRunningTime="2026-07-17 21:53:27.22392921 +0000 UTC m=+2720.983495904"
Jul 17 21:53:28.205682 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:28.205641    2569 generic.go:356] "Generic (PLEG): container finished" podID="ba578e55-e5ca-40e2-85ac-4e090562697d" containerID="42685160851f8d8d887f8908a0503691de1dc53692887474bc7423def1f872b6" exitCode=6
Jul 17 21:53:28.206145 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:28.205729    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8" event={"ID":"ba578e55-e5ca-40e2-85ac-4e090562697d","Type":"ContainerDied","Data":"42685160851f8d8d887f8908a0503691de1dc53692887474bc7423def1f872b6"}
Jul 17 21:53:28.206145 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:28.205804    2569 scope.go:117] "RemoveContainer" containerID="b8c0c1adca11c3b3f4024c5eadab3375287c3bffbd9ebff9ce67397648cb9925"
Jul 17 21:53:28.206145 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:28.206040    2569 scope.go:117] "RemoveContainer" containerID="42685160851f8d8d887f8908a0503691de1dc53692887474bc7423def1f872b6"
Jul 17 21:53:28.206291 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:53:28.206276    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-446f8_odh-ai-gateway-infra(ba578e55-e5ca-40e2-85ac-4e090562697d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8" podUID="ba578e55-e5ca-40e2-85ac-4e090562697d"
Jul 17 21:53:28.207779 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:28.207757    2569 generic.go:356] "Generic (PLEG): container finished" podID="9551a349-e1e5-497a-bf88-4c417bc6390d" containerID="e93b254a1e6de21b02a6199770d90d85de48748cf95e0247a535fe407888c87d" exitCode=6
Jul 17 21:53:28.207866 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:28.207794    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs" event={"ID":"9551a349-e1e5-497a-bf88-4c417bc6390d","Type":"ContainerDied","Data":"e93b254a1e6de21b02a6199770d90d85de48748cf95e0247a535fe407888c87d"}
Jul 17 21:53:28.208068 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:28.208055    2569 scope.go:117] "RemoveContainer" containerID="e93b254a1e6de21b02a6199770d90d85de48748cf95e0247a535fe407888c87d"
Jul 17 21:53:28.208244 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:53:28.208223    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs_odh-ai-gateway-infra(9551a349-e1e5-497a-bf88-4c417bc6390d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs" podUID="9551a349-e1e5-497a-bf88-4c417bc6390d"
Jul 17 21:53:28.217255 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:28.217234    2569 scope.go:117] "RemoveContainer" containerID="d8a9c5c3c2e8b53d21ae692f0b7c6b031e38ad3192ad1b293304fd223b4b18a9"
Jul 17 21:53:28.232063 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:28.232041    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs"]
Jul 17 21:53:29.215324 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:29.215291    2569 scope.go:117] "RemoveContainer" containerID="42685160851f8d8d887f8908a0503691de1dc53692887474bc7423def1f872b6"
Jul 17 21:53:29.215840 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:53:29.215685    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-446f8_odh-ai-gateway-infra(ba578e55-e5ca-40e2-85ac-4e090562697d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8" podUID="ba578e55-e5ca-40e2-85ac-4e090562697d"
Jul 17 21:53:29.366660 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:29.366628    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs"
Jul 17 21:53:29.418754 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:29.418716    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9551a349-e1e5-497a-bf88-4c417bc6390d-maas-api-service-ca\") pod \"9551a349-e1e5-497a-bf88-4c417bc6390d\" (UID: \"9551a349-e1e5-497a-bf88-4c417bc6390d\") "
Jul 17 21:53:29.419064 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:29.419043    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9551a349-e1e5-497a-bf88-4c417bc6390d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9551a349-e1e5-497a-bf88-4c417bc6390d" (UID: "9551a349-e1e5-497a-bf88-4c417bc6390d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:53:29.520324 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:29.520237    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9551a349-e1e5-497a-bf88-4c417bc6390d-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:53:30.224134 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:30.224098    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs"
Jul 17 21:53:30.224605 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:30.224108    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs" event={"ID":"9551a349-e1e5-497a-bf88-4c417bc6390d","Type":"ContainerDied","Data":"3c38aba80f0dc02566c4132d71b254eafa28fbfa517226831133201177812f7c"}
Jul 17 21:53:30.224605 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:30.224238    2569 scope.go:117] "RemoveContainer" containerID="e93b254a1e6de21b02a6199770d90d85de48748cf95e0247a535fe407888c87d"
Jul 17 21:53:30.245785 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:30.245755    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs"]
Jul 17 21:53:30.248868 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:30.248845    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jhxfs"]
Jul 17 21:53:30.656490 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:30.656460    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9551a349-e1e5-497a-bf88-4c417bc6390d" path="/var/lib/kubelet/pods/9551a349-e1e5-497a-bf88-4c417bc6390d/volumes"
Jul 17 21:53:31.520882 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:31.520847    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c39f8651-m2972"]
Jul 17 21:53:31.521269 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:31.521237    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9551a349-e1e5-497a-bf88-4c417bc6390d" containerName="revoke-keys"
Jul 17 21:53:31.521269 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:31.521248    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9551a349-e1e5-497a-bf88-4c417bc6390d" containerName="revoke-keys"
Jul 17 21:53:31.521269 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:31.521265    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9551a349-e1e5-497a-bf88-4c417bc6390d" containerName="revoke-keys"
Jul 17 21:53:31.521376 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:31.521271    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9551a349-e1e5-497a-bf88-4c417bc6390d" containerName="revoke-keys"
Jul 17 21:53:31.521376 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:31.521282    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9551a349-e1e5-497a-bf88-4c417bc6390d" containerName="revoke-keys"
Jul 17 21:53:31.521376 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:31.521288    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9551a349-e1e5-497a-bf88-4c417bc6390d" containerName="revoke-keys"
Jul 17 21:53:31.521376 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:31.521355    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9551a349-e1e5-497a-bf88-4c417bc6390d" containerName="revoke-keys"
Jul 17 21:53:31.521376 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:31.521362    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9551a349-e1e5-497a-bf88-4c417bc6390d" containerName="revoke-keys"
Jul 17 21:53:31.521376 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:31.521368    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9551a349-e1e5-497a-bf88-4c417bc6390d" containerName="revoke-keys"
Jul 17 21:53:31.525818 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:31.525800    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c39f8651-m2972"
Jul 17 21:53:31.531571 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:31.531547    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c39f8651-m2972"]
Jul 17 21:53:31.535521 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:31.535494    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e25088f2-a9c7-414c-951a-919f158b5565-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c39f8651-m2972\" (UID: \"e25088f2-a9c7-414c-951a-919f158b5565\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c39f8651-m2972"
Jul 17 21:53:31.636375 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:31.636338    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e25088f2-a9c7-414c-951a-919f158b5565-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c39f8651-m2972\" (UID: \"e25088f2-a9c7-414c-951a-919f158b5565\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c39f8651-m2972"
Jul 17 21:53:31.637015 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:31.636995    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e25088f2-a9c7-414c-951a-919f158b5565-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c39f8651-m2972\" (UID: \"e25088f2-a9c7-414c-951a-919f158b5565\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c39f8651-m2972"
Jul 17 21:53:31.839290 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:31.839259    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c39f8651-m2972"
Jul 17 21:53:31.972688 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:31.972661    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c39f8651-m2972"]
Jul 17 21:53:31.974785 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:53:31.974757    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25088f2_a9c7_414c_951a_919f158b5565.slice/crio-1b38b5d6786882e972fe7404e70badac20d9404044681f8f822a8b301b6d5d1d WatchSource:0}: Error finding container 1b38b5d6786882e972fe7404e70badac20d9404044681f8f822a8b301b6d5d1d: Status 404 returned error can't find the container with id 1b38b5d6786882e972fe7404e70badac20d9404044681f8f822a8b301b6d5d1d
Jul 17 21:53:32.238665 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:32.238630    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c39f8651-m2972" event={"ID":"e25088f2-a9c7-414c-951a-919f158b5565","Type":"ContainerStarted","Data":"1b38b5d6786882e972fe7404e70badac20d9404044681f8f822a8b301b6d5d1d"}
Jul 17 21:53:33.247455 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:33.247353    2569 generic.go:356] "Generic (PLEG): container finished" podID="e25088f2-a9c7-414c-951a-919f158b5565" containerID="52c84598e78fb311fb6af28540e71981bb182c530ae67af78796aab908ef714c" exitCode=0
Jul 17 21:53:33.247455 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:33.247430    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c39f8651-m2972" event={"ID":"e25088f2-a9c7-414c-951a-919f158b5565","Type":"ContainerDied","Data":"52c84598e78fb311fb6af28540e71981bb182c530ae67af78796aab908ef714c"}
Jul 17 21:53:34.399516 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:34.399492    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c39f8651-m2972"
Jul 17 21:53:34.460888 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:34.460845    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e25088f2-a9c7-414c-951a-919f158b5565-maas-api-service-ca\") pod \"e25088f2-a9c7-414c-951a-919f158b5565\" (UID: \"e25088f2-a9c7-414c-951a-919f158b5565\") "
Jul 17 21:53:34.461259 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:34.461234    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e25088f2-a9c7-414c-951a-919f158b5565-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e25088f2-a9c7-414c-951a-919f158b5565" (UID: "e25088f2-a9c7-414c-951a-919f158b5565"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:53:34.562354 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:34.562246    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e25088f2-a9c7-414c-951a-919f158b5565-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:53:35.260779 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:35.260747    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c39f8651-m2972"
Jul 17 21:53:35.260975 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:35.260786    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c39f8651-m2972" event={"ID":"e25088f2-a9c7-414c-951a-919f158b5565","Type":"ContainerDied","Data":"1b38b5d6786882e972fe7404e70badac20d9404044681f8f822a8b301b6d5d1d"}
Jul 17 21:53:35.260975 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:35.260842    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b38b5d6786882e972fe7404e70badac20d9404044681f8f822a8b301b6d5d1d"
Jul 17 21:53:40.657581 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:40.657551    2569 scope.go:117] "RemoveContainer" containerID="42685160851f8d8d887f8908a0503691de1dc53692887474bc7423def1f872b6"
Jul 17 21:53:41.301589 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:41.301558    2569 generic.go:356] "Generic (PLEG): container finished" podID="ba578e55-e5ca-40e2-85ac-4e090562697d" containerID="5d45bc01038fc0f18ba8181958afa0b1470923cc1ad582e00fbb8a3d9b911d21" exitCode=6
Jul 17 21:53:41.301778 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:41.301601    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8" event={"ID":"ba578e55-e5ca-40e2-85ac-4e090562697d","Type":"ContainerDied","Data":"5d45bc01038fc0f18ba8181958afa0b1470923cc1ad582e00fbb8a3d9b911d21"}
Jul 17 21:53:41.301778 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:41.301653    2569 scope.go:117] "RemoveContainer" containerID="42685160851f8d8d887f8908a0503691de1dc53692887474bc7423def1f872b6"
Jul 17 21:53:41.302000 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:41.301978    2569 scope.go:117] "RemoveContainer" containerID="5d45bc01038fc0f18ba8181958afa0b1470923cc1ad582e00fbb8a3d9b911d21"
Jul 17 21:53:41.302256 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:53:41.302230    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-446f8_odh-ai-gateway-infra(ba578e55-e5ca-40e2-85ac-4e090562697d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8" podUID="ba578e55-e5ca-40e2-85ac-4e090562697d"
Jul 17 21:53:41.997293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:41.997259    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:42.327468 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:42.327353    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8"]
Jul 17 21:53:42.461200 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:42.461169    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8"
Jul 17 21:53:42.544311 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:42.544275    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba578e55-e5ca-40e2-85ac-4e090562697d-maas-api-service-ca\") pod \"ba578e55-e5ca-40e2-85ac-4e090562697d\" (UID: \"ba578e55-e5ca-40e2-85ac-4e090562697d\") "
Jul 17 21:53:42.544727 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:42.544703    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba578e55-e5ca-40e2-85ac-4e090562697d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba578e55-e5ca-40e2-85ac-4e090562697d" (UID: "ba578e55-e5ca-40e2-85ac-4e090562697d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:53:42.645602 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:42.645563    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba578e55-e5ca-40e2-85ac-4e090562697d-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:53:43.315753 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:43.315721    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8"
Jul 17 21:53:43.316196 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:43.315756    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8" event={"ID":"ba578e55-e5ca-40e2-85ac-4e090562697d","Type":"ContainerDied","Data":"4eba6757a325b58591b7db782a6aa333612faef4348808c02563182c34214381"}
Jul 17 21:53:43.316196 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:43.315829    2569 scope.go:117] "RemoveContainer" containerID="5d45bc01038fc0f18ba8181958afa0b1470923cc1ad582e00fbb8a3d9b911d21"
Jul 17 21:53:43.331612 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:43.331585    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8"]
Jul 17 21:53:43.339464 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:43.339434    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-446f8"]
Jul 17 21:53:44.656924 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:44.656889    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba578e55-e5ca-40e2-85ac-4e090562697d" path="/var/lib/kubelet/pods/ba578e55-e5ca-40e2-85ac-4e090562697d/volumes"
Jul 17 21:53:45.912852 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:45.912814    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:53:46.601738 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:46.601689    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:51.588872 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:51.588835    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"]
Jul 17 21:53:51.633950 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:53:51.633915    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:02.158884 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:02.158837    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:02.328692 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:02.328656    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:06.609878 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:06.609819    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m" podUID="55ce6d82-5eb2-4d44-8a9a-44eee751cb8b" containerName="main" containerID="cri-o://372e934ad43ab70431766ab520b9fdcddb49f75c7f9c73728b884ae43119149d" gracePeriod=45
Jul 17 21:54:06.893359 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:06.893330    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:54:06.980831 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:06.980801    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-model-cache\") pod \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") "
Jul 17 21:54:06.980831 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:06.980835    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-tmp-dir\") pod \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") "
Jul 17 21:54:06.981077 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:06.980878    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sjx7\" (UniqueName: \"kubernetes.io/projected/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-kube-api-access-6sjx7\") pod \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") "
Jul 17 21:54:06.981077 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:06.980917    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-tls-certs\") pod \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") "
Jul 17 21:54:06.981077 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:06.980956    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-home\") pod \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") "
Jul 17 21:54:06.981077 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:06.980988    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-dshm\") pod \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\" (UID: \"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b\") "
Jul 17 21:54:06.981294 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:06.981109    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-model-cache" (OuterVolumeSpecName: "model-cache") pod "55ce6d82-5eb2-4d44-8a9a-44eee751cb8b" (UID: "55ce6d82-5eb2-4d44-8a9a-44eee751cb8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:54:06.981294 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:06.981123    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55ce6d82-5eb2-4d44-8a9a-44eee751cb8b" (UID: "55ce6d82-5eb2-4d44-8a9a-44eee751cb8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:54:06.981294 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:06.981221    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-home" (OuterVolumeSpecName: "home") pod "55ce6d82-5eb2-4d44-8a9a-44eee751cb8b" (UID: "55ce6d82-5eb2-4d44-8a9a-44eee751cb8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:54:06.981294 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:06.981276    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-model-cache\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:54:06.981294 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:06.981293    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-tmp-dir\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:54:06.983149 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:06.983122    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-kube-api-access-6sjx7" (OuterVolumeSpecName: "kube-api-access-6sjx7") pod "55ce6d82-5eb2-4d44-8a9a-44eee751cb8b" (UID: "55ce6d82-5eb2-4d44-8a9a-44eee751cb8b"). InnerVolumeSpecName "kube-api-access-6sjx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:54:06.983266 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:06.983127    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55ce6d82-5eb2-4d44-8a9a-44eee751cb8b" (UID: "55ce6d82-5eb2-4d44-8a9a-44eee751cb8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:54:06.983266 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:06.983130    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-dshm" (OuterVolumeSpecName: "dshm") pod "55ce6d82-5eb2-4d44-8a9a-44eee751cb8b" (UID: "55ce6d82-5eb2-4d44-8a9a-44eee751cb8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:54:07.082128 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:07.082098    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-tls-certs\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:54:07.082128 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:07.082126    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-home\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:54:07.082128 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:07.082134    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-dshm\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:54:07.082366 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:07.082143    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sjx7\" (UniqueName: \"kubernetes.io/projected/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b-kube-api-access-6sjx7\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:54:07.455473 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:07.455438    2569 generic.go:356] "Generic (PLEG): container finished" podID="55ce6d82-5eb2-4d44-8a9a-44eee751cb8b" containerID="372e934ad43ab70431766ab520b9fdcddb49f75c7f9c73728b884ae43119149d" exitCode=0
Jul 17 21:54:07.455473 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:07.455476    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"
Jul 17 21:54:07.455727 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:07.455501    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m" event={"ID":"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b","Type":"ContainerDied","Data":"372e934ad43ab70431766ab520b9fdcddb49f75c7f9c73728b884ae43119149d"}
Jul 17 21:54:07.455727 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:07.455533    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m" event={"ID":"55ce6d82-5eb2-4d44-8a9a-44eee751cb8b","Type":"ContainerDied","Data":"fab1731221049b65d962618d8db162fc4564c75b23c4abc63bebfe449452fd2f"}
Jul 17 21:54:07.455727 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:07.455549    2569 scope.go:117] "RemoveContainer" containerID="372e934ad43ab70431766ab520b9fdcddb49f75c7f9c73728b884ae43119149d"
Jul 17 21:54:07.465351 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:07.465329    2569 scope.go:117] "RemoveContainer" containerID="372e934ad43ab70431766ab520b9fdcddb49f75c7f9c73728b884ae43119149d"
Jul 17 21:54:07.465721 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:54:07.465695    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"372e934ad43ab70431766ab520b9fdcddb49f75c7f9c73728b884ae43119149d\": container with ID starting with 372e934ad43ab70431766ab520b9fdcddb49f75c7f9c73728b884ae43119149d not found: ID does not exist" containerID="372e934ad43ab70431766ab520b9fdcddb49f75c7f9c73728b884ae43119149d"
Jul 17 21:54:07.465826 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:07.465732    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"372e934ad43ab70431766ab520b9fdcddb49f75c7f9c73728b884ae43119149d"} err="failed to get container status \"372e934ad43ab70431766ab520b9fdcddb49f75c7f9c73728b884ae43119149d\": rpc error: code = NotFound desc = could not find container \"372e934ad43ab70431766ab520b9fdcddb49f75c7f9c73728b884ae43119149d\": container with ID starting with 372e934ad43ab70431766ab520b9fdcddb49f75c7f9c73728b884ae43119149d not found: ID does not exist"
Jul 17 21:54:07.480465 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:07.480391    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"]
Jul 17 21:54:07.483928 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:07.483900    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c39f8651/e2e-lifecycle-model-c39f8651-kserve-d876f68dc-2ql7m"]
Jul 17 21:54:08.657805 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:08.657767    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55ce6d82-5eb2-4d44-8a9a-44eee751cb8b" path="/var/lib/kubelet/pods/55ce6d82-5eb2-4d44-8a9a-44eee751cb8b/volumes"
Jul 17 21:54:13.838091 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:13.838053    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:13.902309 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:13.902275    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:15.351091 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.351052    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx"]
Jul 17 21:54:15.351519 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.351480    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba578e55-e5ca-40e2-85ac-4e090562697d" containerName="revoke-keys"
Jul 17 21:54:15.351519 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.351492    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba578e55-e5ca-40e2-85ac-4e090562697d" containerName="revoke-keys"
Jul 17 21:54:15.351519 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.351504    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba578e55-e5ca-40e2-85ac-4e090562697d" containerName="revoke-keys"
Jul 17 21:54:15.351519 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.351512    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba578e55-e5ca-40e2-85ac-4e090562697d" containerName="revoke-keys"
Jul 17 21:54:15.351668 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.351526    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e25088f2-a9c7-414c-951a-919f158b5565" containerName="revoke-keys"
Jul 17 21:54:15.351668 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.351531    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e25088f2-a9c7-414c-951a-919f158b5565" containerName="revoke-keys"
Jul 17 21:54:15.351668 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.351542    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55ce6d82-5eb2-4d44-8a9a-44eee751cb8b" containerName="main"
Jul 17 21:54:15.351668 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.351548    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="55ce6d82-5eb2-4d44-8a9a-44eee751cb8b" containerName="main"
Jul 17 21:54:15.351668 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.351634    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e25088f2-a9c7-414c-951a-919f158b5565" containerName="revoke-keys"
Jul 17 21:54:15.351668 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.351644    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba578e55-e5ca-40e2-85ac-4e090562697d" containerName="revoke-keys"
Jul 17 21:54:15.351668 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.351652    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="55ce6d82-5eb2-4d44-8a9a-44eee751cb8b" containerName="main"
Jul 17 21:54:15.351668 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.351669    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba578e55-e5ca-40e2-85ac-4e090562697d" containerName="revoke-keys"
Jul 17 21:54:15.356111 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.356091    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx"
Jul 17 21:54:15.358800 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.358776    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:54:15.359231 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.358785    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:54:15.362281 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.362259    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx"]
Jul 17 21:54:15.367635 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.367609    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c6ff13-b368-4675-b11d-7f0f8341c9ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx\" (UID: \"30c6ff13-b368-4675-b11d-7f0f8341c9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx"
Jul 17 21:54:15.468116 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.468077    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c6ff13-b368-4675-b11d-7f0f8341c9ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx\" (UID: \"30c6ff13-b368-4675-b11d-7f0f8341c9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx"
Jul 17 21:54:15.468767 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.468747    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c6ff13-b368-4675-b11d-7f0f8341c9ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx\" (UID: \"30c6ff13-b368-4675-b11d-7f0f8341c9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx"
Jul 17 21:54:15.668524 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.668429    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx"
Jul 17 21:54:15.807214 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:15.807191    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx"]
Jul 17 21:54:15.808907 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:54:15.808878    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c6ff13_b368_4675_b11d_7f0f8341c9ab.slice/crio-aee18564071baf57f4da30de1b32d10418eec2c628d5fbfba551fb40e0795460 WatchSource:0}: Error finding container aee18564071baf57f4da30de1b32d10418eec2c628d5fbfba551fb40e0795460: Status 404 returned error can't find the container with id aee18564071baf57f4da30de1b32d10418eec2c628d5fbfba551fb40e0795460
Jul 17 21:54:16.514017 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:16.513985    2569 generic.go:356] "Generic (PLEG): container finished" podID="30c6ff13-b368-4675-b11d-7f0f8341c9ab" containerID="55afd6f3d68613ef86c15aad64ea60dc48450f9fc801006f6289d5fd3c501c11" exitCode=6
Jul 17 21:54:16.514462 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:16.514024    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx" event={"ID":"30c6ff13-b368-4675-b11d-7f0f8341c9ab","Type":"ContainerDied","Data":"55afd6f3d68613ef86c15aad64ea60dc48450f9fc801006f6289d5fd3c501c11"}
Jul 17 21:54:16.514462 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:16.514085    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx" event={"ID":"30c6ff13-b368-4675-b11d-7f0f8341c9ab","Type":"ContainerStarted","Data":"aee18564071baf57f4da30de1b32d10418eec2c628d5fbfba551fb40e0795460"}
Jul 17 21:54:16.514462 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:16.514303    2569 scope.go:117] "RemoveContainer" containerID="55afd6f3d68613ef86c15aad64ea60dc48450f9fc801006f6289d5fd3c501c11"
Jul 17 21:54:17.521053 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:17.521015    2569 generic.go:356] "Generic (PLEG): container finished" podID="30c6ff13-b368-4675-b11d-7f0f8341c9ab" containerID="4002b545b23e528cf4c79eb4eadfe0207c2163b09faab8aaf5abfa11251fc19f" exitCode=6
Jul 17 21:54:17.521540 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:17.521099    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx" event={"ID":"30c6ff13-b368-4675-b11d-7f0f8341c9ab","Type":"ContainerDied","Data":"4002b545b23e528cf4c79eb4eadfe0207c2163b09faab8aaf5abfa11251fc19f"}
Jul 17 21:54:17.521540 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:17.521162    2569 scope.go:117] "RemoveContainer" containerID="55afd6f3d68613ef86c15aad64ea60dc48450f9fc801006f6289d5fd3c501c11"
Jul 17 21:54:17.521540 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:17.521362    2569 scope.go:117] "RemoveContainer" containerID="4002b545b23e528cf4c79eb4eadfe0207c2163b09faab8aaf5abfa11251fc19f"
Jul 17 21:54:17.521683 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:54:17.521593    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx_odh-ai-gateway-infra(30c6ff13-b368-4675-b11d-7f0f8341c9ab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx" podUID="30c6ff13-b368-4675-b11d-7f0f8341c9ab"
Jul 17 21:54:18.528748 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:18.528718    2569 scope.go:117] "RemoveContainer" containerID="4002b545b23e528cf4c79eb4eadfe0207c2163b09faab8aaf5abfa11251fc19f"
Jul 17 21:54:18.529126 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:54:18.528938    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx_odh-ai-gateway-infra(30c6ff13-b368-4675-b11d-7f0f8341c9ab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx" podUID="30c6ff13-b368-4675-b11d-7f0f8341c9ab"
Jul 17 21:54:22.091879 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:22.091847    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:23.363065 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:23.363024    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:28.992152 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:28.992113    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:29.052940 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:29.052907    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:31.651131 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:31.651102    2569 scope.go:117] "RemoveContainer" containerID="4002b545b23e528cf4c79eb4eadfe0207c2163b09faab8aaf5abfa11251fc19f"
Jul 17 21:54:32.612592 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:32.612499    2569 generic.go:356] "Generic (PLEG): container finished" podID="30c6ff13-b368-4675-b11d-7f0f8341c9ab" containerID="f9c702da14a30ccee733d67b47ab7f980893dcceb09dc25d9f7a359824c19559" exitCode=6
Jul 17 21:54:32.612592 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:32.612574    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx" event={"ID":"30c6ff13-b368-4675-b11d-7f0f8341c9ab","Type":"ContainerDied","Data":"f9c702da14a30ccee733d67b47ab7f980893dcceb09dc25d9f7a359824c19559"}
Jul 17 21:54:32.612826 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:32.612631    2569 scope.go:117] "RemoveContainer" containerID="4002b545b23e528cf4c79eb4eadfe0207c2163b09faab8aaf5abfa11251fc19f"
Jul 17 21:54:32.612992 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:32.612967    2569 scope.go:117] "RemoveContainer" containerID="f9c702da14a30ccee733d67b47ab7f980893dcceb09dc25d9f7a359824c19559"
Jul 17 21:54:32.613218 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:54:32.613189    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx_odh-ai-gateway-infra(30c6ff13-b368-4675-b11d-7f0f8341c9ab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx" podUID="30c6ff13-b368-4675-b11d-7f0f8341c9ab"
Jul 17 21:54:33.638366 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:33.638330    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx"]
Jul 17 21:54:33.777498 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:33.777469    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx"
Jul 17 21:54:33.813096 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:33.813061    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c6ff13-b368-4675-b11d-7f0f8341c9ab-maas-api-service-ca\") pod \"30c6ff13-b368-4675-b11d-7f0f8341c9ab\" (UID: \"30c6ff13-b368-4675-b11d-7f0f8341c9ab\") "
Jul 17 21:54:33.813439 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:33.813393    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c6ff13-b368-4675-b11d-7f0f8341c9ab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "30c6ff13-b368-4675-b11d-7f0f8341c9ab" (UID: "30c6ff13-b368-4675-b11d-7f0f8341c9ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:54:33.914394 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:33.914308    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c6ff13-b368-4675-b11d-7f0f8341c9ab-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:54:34.627132 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:34.627090    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx" event={"ID":"30c6ff13-b368-4675-b11d-7f0f8341c9ab","Type":"ContainerDied","Data":"aee18564071baf57f4da30de1b32d10418eec2c628d5fbfba551fb40e0795460"}
Jul 17 21:54:34.627132 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:34.627109    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx"
Jul 17 21:54:34.627389 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:34.627165    2569 scope.go:117] "RemoveContainer" containerID="f9c702da14a30ccee733d67b47ab7f980893dcceb09dc25d9f7a359824c19559"
Jul 17 21:54:34.657606 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:34.657574    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx"]
Jul 17 21:54:34.658762 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:34.658743    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-ncxsx"]
Jul 17 21:54:36.656887 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:36.656853    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30c6ff13-b368-4675-b11d-7f0f8341c9ab" path="/var/lib/kubelet/pods/30c6ff13-b368-4675-b11d-7f0f8341c9ab/volumes"
Jul 17 21:54:38.786829 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:38.786791    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:38.999729 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:38.999693    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:40.043804 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:40.043770    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:40.142073 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:40.142045    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:40.242166 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:40.242121    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:40.923789 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:40.923749    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:41.026758 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:41.026725    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:41.126456 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:41.126384    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:42.414357 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:42.414324    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:43.024068 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:43.024034    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:54:48.278671 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:48.278633    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:48.340418 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:48.340374    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:48.942395 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:48.942366    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:54.729890 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:54.729851    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:54.834644 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:54.834610    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:54.931647 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:54.931612    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:56.113666 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:56.113628    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:54:59.506139 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:59.506102    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj"]
Jul 17 21:54:59.506543 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:59.506494    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c6ff13-b368-4675-b11d-7f0f8341c9ab" containerName="revoke-keys"
Jul 17 21:54:59.506543 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:59.506506    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c6ff13-b368-4675-b11d-7f0f8341c9ab" containerName="revoke-keys"
Jul 17 21:54:59.506543 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:59.506514    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c6ff13-b368-4675-b11d-7f0f8341c9ab" containerName="revoke-keys"
Jul 17 21:54:59.506543 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:59.506519    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c6ff13-b368-4675-b11d-7f0f8341c9ab" containerName="revoke-keys"
Jul 17 21:54:59.506543 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:59.506538    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba578e55-e5ca-40e2-85ac-4e090562697d" containerName="revoke-keys"
Jul 17 21:54:59.506543 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:59.506543    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba578e55-e5ca-40e2-85ac-4e090562697d" containerName="revoke-keys"
Jul 17 21:54:59.506738 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:59.506608    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c6ff13-b368-4675-b11d-7f0f8341c9ab" containerName="revoke-keys"
Jul 17 21:54:59.506738 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:59.506620    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c6ff13-b368-4675-b11d-7f0f8341c9ab" containerName="revoke-keys"
Jul 17 21:54:59.506738 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:59.506625    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c6ff13-b368-4675-b11d-7f0f8341c9ab" containerName="revoke-keys"
Jul 17 21:54:59.506738 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:59.506632    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba578e55-e5ca-40e2-85ac-4e090562697d" containerName="revoke-keys"
Jul 17 21:54:59.510305 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:59.510287    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj"
Jul 17 21:54:59.512469 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:59.512442    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:54:59.512615 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:59.512495    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:54:59.518112 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:59.518074    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj"]
Jul 17 21:54:59.629084 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:59.629039    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a93c852d-2387-4366-8e3a-a345164a1f09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj\" (UID: \"a93c852d-2387-4366-8e3a-a345164a1f09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj"
Jul 17 21:54:59.730448 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:59.730383    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a93c852d-2387-4366-8e3a-a345164a1f09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj\" (UID: \"a93c852d-2387-4366-8e3a-a345164a1f09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj"
Jul 17 21:54:59.731041 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:59.731022    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a93c852d-2387-4366-8e3a-a345164a1f09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj\" (UID: \"a93c852d-2387-4366-8e3a-a345164a1f09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj"
Jul 17 21:54:59.822755 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:59.822678    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj"
Jul 17 21:54:59.956352 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:54:59.956327    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj"]
Jul 17 21:54:59.957580 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:54:59.957551    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93c852d_2387_4366_8e3a_a345164a1f09.slice/crio-cc2110a65b21a682ace8cd92c678b33640604ae00ede21310828ad138f03153d WatchSource:0}: Error finding container cc2110a65b21a682ace8cd92c678b33640604ae00ede21310828ad138f03153d: Status 404 returned error can't find the container with id cc2110a65b21a682ace8cd92c678b33640604ae00ede21310828ad138f03153d
Jul 17 21:55:00.782518 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:00.782482    2569 generic.go:356] "Generic (PLEG): container finished" podID="a93c852d-2387-4366-8e3a-a345164a1f09" containerID="c05c558498e6f15620f5120eeae778b8260f37d1bb23b66f22eba38607d5151a" exitCode=6
Jul 17 21:55:00.782904 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:00.782513    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj" event={"ID":"a93c852d-2387-4366-8e3a-a345164a1f09","Type":"ContainerDied","Data":"c05c558498e6f15620f5120eeae778b8260f37d1bb23b66f22eba38607d5151a"}
Jul 17 21:55:00.782904 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:00.782567    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj" event={"ID":"a93c852d-2387-4366-8e3a-a345164a1f09","Type":"ContainerStarted","Data":"cc2110a65b21a682ace8cd92c678b33640604ae00ede21310828ad138f03153d"}
Jul 17 21:55:00.782904 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:00.782755    2569 scope.go:117] "RemoveContainer" containerID="c05c558498e6f15620f5120eeae778b8260f37d1bb23b66f22eba38607d5151a"
Jul 17 21:55:01.790004 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:01.789963    2569 generic.go:356] "Generic (PLEG): container finished" podID="a93c852d-2387-4366-8e3a-a345164a1f09" containerID="d409d8101e9ee5cfe69c021eea05a170e1c3145caeda4720b515e120e7fc813a" exitCode=6
Jul 17 21:55:01.790437 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:01.790042    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj" event={"ID":"a93c852d-2387-4366-8e3a-a345164a1f09","Type":"ContainerDied","Data":"d409d8101e9ee5cfe69c021eea05a170e1c3145caeda4720b515e120e7fc813a"}
Jul 17 21:55:01.790437 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:01.790103    2569 scope.go:117] "RemoveContainer" containerID="c05c558498e6f15620f5120eeae778b8260f37d1bb23b66f22eba38607d5151a"
Jul 17 21:55:01.790437 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:01.790312    2569 scope.go:117] "RemoveContainer" containerID="d409d8101e9ee5cfe69c021eea05a170e1c3145caeda4720b515e120e7fc813a"
Jul 17 21:55:01.790561 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:55:01.790538    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj_odh-ai-gateway-infra(a93c852d-2387-4366-8e3a-a345164a1f09)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj" podUID="a93c852d-2387-4366-8e3a-a345164a1f09"
Jul 17 21:55:02.801130 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:02.801103    2569 scope.go:117] "RemoveContainer" containerID="d409d8101e9ee5cfe69c021eea05a170e1c3145caeda4720b515e120e7fc813a"
Jul 17 21:55:02.801570 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:55:02.801297    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj_odh-ai-gateway-infra(a93c852d-2387-4366-8e3a-a345164a1f09)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj" podUID="a93c852d-2387-4366-8e3a-a345164a1f09"
Jul 17 21:55:05.427128 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:05.427086    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:05.764124 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:05.764040    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:13.374684 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:13.374650    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:14.650967 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:14.650933    2569 scope.go:117] "RemoveContainer" containerID="d409d8101e9ee5cfe69c021eea05a170e1c3145caeda4720b515e120e7fc813a"
Jul 17 21:55:15.879757 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:15.879726    2569 generic.go:356] "Generic (PLEG): container finished" podID="a93c852d-2387-4366-8e3a-a345164a1f09" containerID="2dfafba20b8bb5266dbb45f840a10024974104f60b0fce71d27b404bc16d8133" exitCode=6
Jul 17 21:55:15.880150 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:15.879791    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj" event={"ID":"a93c852d-2387-4366-8e3a-a345164a1f09","Type":"ContainerDied","Data":"2dfafba20b8bb5266dbb45f840a10024974104f60b0fce71d27b404bc16d8133"}
Jul 17 21:55:15.880150 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:15.879842    2569 scope.go:117] "RemoveContainer" containerID="d409d8101e9ee5cfe69c021eea05a170e1c3145caeda4720b515e120e7fc813a"
Jul 17 21:55:15.880150 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:15.880139    2569 scope.go:117] "RemoveContainer" containerID="2dfafba20b8bb5266dbb45f840a10024974104f60b0fce71d27b404bc16d8133"
Jul 17 21:55:15.880337 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:55:15.880316    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj_odh-ai-gateway-infra(a93c852d-2387-4366-8e3a-a345164a1f09)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj" podUID="a93c852d-2387-4366-8e3a-a345164a1f09"
Jul 17 21:55:16.904948 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:16.904917    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj"]
Jul 17 21:55:17.040023 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:17.039998    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj"
Jul 17 21:55:17.181034 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:17.180997    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a93c852d-2387-4366-8e3a-a345164a1f09-maas-api-service-ca\") pod \"a93c852d-2387-4366-8e3a-a345164a1f09\" (UID: \"a93c852d-2387-4366-8e3a-a345164a1f09\") "
Jul 17 21:55:17.181380 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:17.181357    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a93c852d-2387-4366-8e3a-a345164a1f09-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a93c852d-2387-4366-8e3a-a345164a1f09" (UID: "a93c852d-2387-4366-8e3a-a345164a1f09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:55:17.281741 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:17.281702    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a93c852d-2387-4366-8e3a-a345164a1f09-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:55:17.894286 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:17.894248    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj" event={"ID":"a93c852d-2387-4366-8e3a-a345164a1f09","Type":"ContainerDied","Data":"cc2110a65b21a682ace8cd92c678b33640604ae00ede21310828ad138f03153d"}
Jul 17 21:55:17.894507 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:17.894307    2569 scope.go:117] "RemoveContainer" containerID="2dfafba20b8bb5266dbb45f840a10024974104f60b0fce71d27b404bc16d8133"
Jul 17 21:55:17.894507 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:17.894256    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj"
Jul 17 21:55:17.917820 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:17.917793    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj"]
Jul 17 21:55:17.923907 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:17.923885    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-vw7cj"]
Jul 17 21:55:18.656848 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:18.656815    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a93c852d-2387-4366-8e3a-a345164a1f09" path="/var/lib/kubelet/pods/a93c852d-2387-4366-8e3a-a345164a1f09/volumes"
Jul 17 21:55:26.767322 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:26.767282    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:32.447862 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:32.447829    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:32.521576 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:32.521545    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:32.736353 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:32.736275    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:32.836496 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:32.836464    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:33.187905 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:33.187876    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:33.287953 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:33.287919    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:35.858044 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:35.857980    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:35.934732 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:35.934665    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:36.055955 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:36.055909    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:36.252052 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:36.251969    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:36.654964 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:36.654931    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:36.760168 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:36.760129    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:37.665811 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:37.665772    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2"]
Jul 17 21:55:37.666348 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:37.666332    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93c852d-2387-4366-8e3a-a345164a1f09" containerName="revoke-keys"
Jul 17 21:55:37.666393 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:37.666353    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93c852d-2387-4366-8e3a-a345164a1f09" containerName="revoke-keys"
Jul 17 21:55:37.666393 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:37.666380    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93c852d-2387-4366-8e3a-a345164a1f09" containerName="revoke-keys"
Jul 17 21:55:37.666393 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:37.666389    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93c852d-2387-4366-8e3a-a345164a1f09" containerName="revoke-keys"
Jul 17 21:55:37.666522 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:37.666415    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93c852d-2387-4366-8e3a-a345164a1f09" containerName="revoke-keys"
Jul 17 21:55:37.666522 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:37.666424    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93c852d-2387-4366-8e3a-a345164a1f09" containerName="revoke-keys"
Jul 17 21:55:37.666522 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:37.666437    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c6ff13-b368-4675-b11d-7f0f8341c9ab" containerName="revoke-keys"
Jul 17 21:55:37.666522 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:37.666445    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c6ff13-b368-4675-b11d-7f0f8341c9ab" containerName="revoke-keys"
Jul 17 21:55:37.666648 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:37.666556    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a93c852d-2387-4366-8e3a-a345164a1f09" containerName="revoke-keys"
Jul 17 21:55:37.666648 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:37.666567    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a93c852d-2387-4366-8e3a-a345164a1f09" containerName="revoke-keys"
Jul 17 21:55:37.666648 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:37.666577    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a93c852d-2387-4366-8e3a-a345164a1f09" containerName="revoke-keys"
Jul 17 21:55:37.671360 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:37.671340    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2"
Jul 17 21:55:37.673599 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:37.673572    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:55:37.673726 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:37.673581    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:55:37.676344 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:37.676322    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2"]
Jul 17 21:55:37.757695 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:37.757662    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c748e2-fd92-4660-a253-eefeb9b45126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2\" (UID: \"74c748e2-fd92-4660-a253-eefeb9b45126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2"
Jul 17 21:55:37.859110 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:37.859072    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c748e2-fd92-4660-a253-eefeb9b45126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2\" (UID: \"74c748e2-fd92-4660-a253-eefeb9b45126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2"
Jul 17 21:55:37.859687 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:37.859661    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c748e2-fd92-4660-a253-eefeb9b45126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2\" (UID: \"74c748e2-fd92-4660-a253-eefeb9b45126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2"
Jul 17 21:55:37.984294 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:37.984211    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2"
Jul 17 21:55:38.117053 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:38.117024    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2"]
Jul 17 21:55:38.118585 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:55:38.118547    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c748e2_fd92_4660_a253_eefeb9b45126.slice/crio-3c3f78e8d677baa2068fef681b73952ec496d97fdc23304021a917222115f13e WatchSource:0}: Error finding container 3c3f78e8d677baa2068fef681b73952ec496d97fdc23304021a917222115f13e: Status 404 returned error can't find the container with id 3c3f78e8d677baa2068fef681b73952ec496d97fdc23304021a917222115f13e
Jul 17 21:55:39.022443 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:39.022333    2569 generic.go:356] "Generic (PLEG): container finished" podID="74c748e2-fd92-4660-a253-eefeb9b45126" containerID="02a4ae1e9ded95138aeda9fe16c9c09036fef28671226a27f2f19d2ed7ef06c2" exitCode=6
Jul 17 21:55:39.022443 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:39.022382    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2" event={"ID":"74c748e2-fd92-4660-a253-eefeb9b45126","Type":"ContainerDied","Data":"02a4ae1e9ded95138aeda9fe16c9c09036fef28671226a27f2f19d2ed7ef06c2"}
Jul 17 21:55:39.022905 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:39.022463    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2" event={"ID":"74c748e2-fd92-4660-a253-eefeb9b45126","Type":"ContainerStarted","Data":"3c3f78e8d677baa2068fef681b73952ec496d97fdc23304021a917222115f13e"}
Jul 17 21:55:39.022905 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:39.022682    2569 scope.go:117] "RemoveContainer" containerID="02a4ae1e9ded95138aeda9fe16c9c09036fef28671226a27f2f19d2ed7ef06c2"
Jul 17 21:55:40.029600 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:40.029509    2569 generic.go:356] "Generic (PLEG): container finished" podID="74c748e2-fd92-4660-a253-eefeb9b45126" containerID="ff98050d30188285c4e72b9e3f3d0a059846d94bf0e036319f7ef775557fb9af" exitCode=6
Jul 17 21:55:40.030016 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:40.029589    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2" event={"ID":"74c748e2-fd92-4660-a253-eefeb9b45126","Type":"ContainerDied","Data":"ff98050d30188285c4e72b9e3f3d0a059846d94bf0e036319f7ef775557fb9af"}
Jul 17 21:55:40.030016 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:40.029675    2569 scope.go:117] "RemoveContainer" containerID="02a4ae1e9ded95138aeda9fe16c9c09036fef28671226a27f2f19d2ed7ef06c2"
Jul 17 21:55:40.030016 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:40.029906    2569 scope.go:117] "RemoveContainer" containerID="ff98050d30188285c4e72b9e3f3d0a059846d94bf0e036319f7ef775557fb9af"
Jul 17 21:55:40.030177 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:55:40.030137    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2_odh-ai-gateway-infra(74c748e2-fd92-4660-a253-eefeb9b45126)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2" podUID="74c748e2-fd92-4660-a253-eefeb9b45126"
Jul 17 21:55:40.340921 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:40.340894    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:40.653137 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:40.653053    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:40.947726 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:40.947642    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:41.042170 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:41.042147    2569 scope.go:117] "RemoveContainer" containerID="ff98050d30188285c4e72b9e3f3d0a059846d94bf0e036319f7ef775557fb9af"
Jul 17 21:55:41.042566 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:55:41.042371    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2_odh-ai-gateway-infra(74c748e2-fd92-4660-a253-eefeb9b45126)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2" podUID="74c748e2-fd92-4660-a253-eefeb9b45126"
Jul 17 21:55:41.431528 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:41.431488    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:43.989740 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:43.989705    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:44.172972 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:44.172945    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:44.320911 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:44.320827    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:44.667999 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:44.667963    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:55:44.994293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:44.994218    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:45.250552 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:45.250464    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:45.362907 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:45.362871    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:48.343585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:48.343553    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:49.548809 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:49.548772    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h"]
Jul 17 21:55:49.552393 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:49.552375    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h"
Jul 17 21:55:49.560301 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:49.560272    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h"]
Jul 17 21:55:49.667040 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:49.667000    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e380cc23-429a-47c6-8859-f829ccc9ba1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h\" (UID: \"e380cc23-429a-47c6-8859-f829ccc9ba1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h"
Jul 17 21:55:49.768415 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:49.768369    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e380cc23-429a-47c6-8859-f829ccc9ba1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h\" (UID: \"e380cc23-429a-47c6-8859-f829ccc9ba1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h"
Jul 17 21:55:49.769017 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:49.768995    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e380cc23-429a-47c6-8859-f829ccc9ba1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h\" (UID: \"e380cc23-429a-47c6-8859-f829ccc9ba1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h"
Jul 17 21:55:49.865410 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:49.865371    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h"
Jul 17 21:55:50.002519 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:50.002492    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h"]
Jul 17 21:55:50.004635 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:55:50.004607    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode380cc23_429a_47c6_8859_f829ccc9ba1e.slice/crio-9215fd662db7e65e705ae614abf1b51480ade2aca5ef3804dfd14bd7e1e55c40 WatchSource:0}: Error finding container 9215fd662db7e65e705ae614abf1b51480ade2aca5ef3804dfd14bd7e1e55c40: Status 404 returned error can't find the container with id 9215fd662db7e65e705ae614abf1b51480ade2aca5ef3804dfd14bd7e1e55c40
Jul 17 21:55:50.097300 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:50.097267    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h" event={"ID":"e380cc23-429a-47c6-8859-f829ccc9ba1e","Type":"ContainerStarted","Data":"9215fd662db7e65e705ae614abf1b51480ade2aca5ef3804dfd14bd7e1e55c40"}
Jul 17 21:55:51.109151 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:51.109118    2569 generic.go:356] "Generic (PLEG): container finished" podID="e380cc23-429a-47c6-8859-f829ccc9ba1e" containerID="ddaf5860b695325a1efb98377fab263bc28880c938833b62368b9aaa721fc7c0" exitCode=6
Jul 17 21:55:51.109630 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:51.109205    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h" event={"ID":"e380cc23-429a-47c6-8859-f829ccc9ba1e","Type":"ContainerDied","Data":"ddaf5860b695325a1efb98377fab263bc28880c938833b62368b9aaa721fc7c0"}
Jul 17 21:55:51.109630 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:51.109353    2569 scope.go:117] "RemoveContainer" containerID="ddaf5860b695325a1efb98377fab263bc28880c938833b62368b9aaa721fc7c0"
Jul 17 21:55:52.117220 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:52.117187    2569 generic.go:356] "Generic (PLEG): container finished" podID="e380cc23-429a-47c6-8859-f829ccc9ba1e" containerID="61243ba0c5e609f6f84f4b7fccf16f968917913be31ea93d11f5b05b3d81e315" exitCode=6
Jul 17 21:55:52.117636 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:52.117270    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h" event={"ID":"e380cc23-429a-47c6-8859-f829ccc9ba1e","Type":"ContainerDied","Data":"61243ba0c5e609f6f84f4b7fccf16f968917913be31ea93d11f5b05b3d81e315"}
Jul 17 21:55:52.117636 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:52.117328    2569 scope.go:117] "RemoveContainer" containerID="ddaf5860b695325a1efb98377fab263bc28880c938833b62368b9aaa721fc7c0"
Jul 17 21:55:52.117636 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:52.117476    2569 scope.go:117] "RemoveContainer" containerID="61243ba0c5e609f6f84f4b7fccf16f968917913be31ea93d11f5b05b3d81e315"
Jul 17 21:55:52.117763 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:55:52.117678    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h_odh-ai-gateway-infra(e380cc23-429a-47c6-8859-f829ccc9ba1e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h" podUID="e380cc23-429a-47c6-8859-f829ccc9ba1e"
Jul 17 21:55:53.125153 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:53.125131    2569 scope.go:117] "RemoveContainer" containerID="61243ba0c5e609f6f84f4b7fccf16f968917913be31ea93d11f5b05b3d81e315"
Jul 17 21:55:53.125535 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:55:53.125316    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h_odh-ai-gateway-infra(e380cc23-429a-47c6-8859-f829ccc9ba1e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h" podUID="e380cc23-429a-47c6-8859-f829ccc9ba1e"
Jul 17 21:55:53.650300 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:53.650270    2569 scope.go:117] "RemoveContainer" containerID="ff98050d30188285c4e72b9e3f3d0a059846d94bf0e036319f7ef775557fb9af"
Jul 17 21:55:54.133417 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:54.133370    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2" event={"ID":"74c748e2-fd92-4660-a253-eefeb9b45126","Type":"ContainerStarted","Data":"123f0e897715a2db8cd42905265ab9dbbc7f356806723cd5f108291e2bae109e"}
Jul 17 21:55:54.148302 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:54.148252    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2" podStartSLOduration=17.148237362 podStartE2EDuration="17.148237362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:55:54.14697072 +0000 UTC m=+2867.906537413" watchObservedRunningTime="2026-07-17 21:55:54.148237362 +0000 UTC m=+2867.907804057"
Jul 17 21:55:55.140574 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:55.140542    2569 generic.go:356] "Generic (PLEG): container finished" podID="74c748e2-fd92-4660-a253-eefeb9b45126" containerID="123f0e897715a2db8cd42905265ab9dbbc7f356806723cd5f108291e2bae109e" exitCode=6
Jul 17 21:55:55.141005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:55.140618    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2" event={"ID":"74c748e2-fd92-4660-a253-eefeb9b45126","Type":"ContainerDied","Data":"123f0e897715a2db8cd42905265ab9dbbc7f356806723cd5f108291e2bae109e"}
Jul 17 21:55:55.141005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:55.140673    2569 scope.go:117] "RemoveContainer" containerID="ff98050d30188285c4e72b9e3f3d0a059846d94bf0e036319f7ef775557fb9af"
Jul 17 21:55:55.141005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:55.140974    2569 scope.go:117] "RemoveContainer" containerID="123f0e897715a2db8cd42905265ab9dbbc7f356806723cd5f108291e2bae109e"
Jul 17 21:55:55.141217 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:55:55.141191    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2_odh-ai-gateway-infra(74c748e2-fd92-4660-a253-eefeb9b45126)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2" podUID="74c748e2-fd92-4660-a253-eefeb9b45126"
Jul 17 21:55:55.164780 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:55.164743    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2"]
Jul 17 21:55:55.436961 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:55.436857    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq"]
Jul 17 21:55:55.441671 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:55.441646    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq"
Jul 17 21:55:55.448429 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:55.448385    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq"]
Jul 17 21:55:55.518793 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:55.518765    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/555c8dcd-c3d3-449e-a73b-587e1fad896b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq\" (UID: \"555c8dcd-c3d3-449e-a73b-587e1fad896b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq"
Jul 17 21:55:55.619729 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:55.619664    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/555c8dcd-c3d3-449e-a73b-587e1fad896b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq\" (UID: \"555c8dcd-c3d3-449e-a73b-587e1fad896b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq"
Jul 17 21:55:55.620305 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:55.620285    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/555c8dcd-c3d3-449e-a73b-587e1fad896b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq\" (UID: \"555c8dcd-c3d3-449e-a73b-587e1fad896b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq"
Jul 17 21:55:55.754529 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:55.754450    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq"
Jul 17 21:55:55.903035 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:55.903005    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq"]
Jul 17 21:55:55.904187 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:55:55.904157    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555c8dcd_c3d3_449e_a73b_587e1fad896b.slice/crio-c832b29ee9dd272d1480d1268afda45cf21099ce04f6ec0e05f6da02b2d3952d WatchSource:0}: Error finding container c832b29ee9dd272d1480d1268afda45cf21099ce04f6ec0e05f6da02b2d3952d: Status 404 returned error can't find the container with id c832b29ee9dd272d1480d1268afda45cf21099ce04f6ec0e05f6da02b2d3952d
Jul 17 21:55:56.116028 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:56.115999    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:55:56.149986 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:56.149960    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq" event={"ID":"555c8dcd-c3d3-449e-a73b-587e1fad896b","Type":"ContainerStarted","Data":"389086bb7301916abad1a385f6902b4651b51327fb2fdea83b2adbc996453c30"}
Jul 17 21:55:56.150296 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:56.150012    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq" event={"ID":"555c8dcd-c3d3-449e-a73b-587e1fad896b","Type":"ContainerStarted","Data":"c832b29ee9dd272d1480d1268afda45cf21099ce04f6ec0e05f6da02b2d3952d"}
Jul 17 21:55:56.175355 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:56.175291    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq" podStartSLOduration=1.175274534 podStartE2EDuration="1.175274534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:55:56.173921238 +0000 UTC m=+2869.933487932" watchObservedRunningTime="2026-07-17 21:55:56.175274534 +0000 UTC m=+2869.934841261"
Jul 17 21:55:56.284898 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:56.284860    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2"
Jul 17 21:55:56.430231 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:56.430204    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c748e2-fd92-4660-a253-eefeb9b45126-maas-api-service-ca\") pod \"74c748e2-fd92-4660-a253-eefeb9b45126\" (UID: \"74c748e2-fd92-4660-a253-eefeb9b45126\") "
Jul 17 21:55:56.430585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:56.430563    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c748e2-fd92-4660-a253-eefeb9b45126-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74c748e2-fd92-4660-a253-eefeb9b45126" (UID: "74c748e2-fd92-4660-a253-eefeb9b45126"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:55:56.530935 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:56.530906    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c748e2-fd92-4660-a253-eefeb9b45126-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:55:57.157028 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:57.157002    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2"
Jul 17 21:55:57.157028 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:57.157001    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2" event={"ID":"74c748e2-fd92-4660-a253-eefeb9b45126","Type":"ContainerDied","Data":"3c3f78e8d677baa2068fef681b73952ec496d97fdc23304021a917222115f13e"}
Jul 17 21:55:57.157507 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:57.157078    2569 scope.go:117] "RemoveContainer" containerID="123f0e897715a2db8cd42905265ab9dbbc7f356806723cd5f108291e2bae109e"
Jul 17 21:55:57.158727 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:57.158704    2569 generic.go:356] "Generic (PLEG): container finished" podID="555c8dcd-c3d3-449e-a73b-587e1fad896b" containerID="389086bb7301916abad1a385f6902b4651b51327fb2fdea83b2adbc996453c30" exitCode=6
Jul 17 21:55:57.158855 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:57.158834    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq" event={"ID":"555c8dcd-c3d3-449e-a73b-587e1fad896b","Type":"ContainerDied","Data":"389086bb7301916abad1a385f6902b4651b51327fb2fdea83b2adbc996453c30"}
Jul 17 21:55:57.159175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:57.159154    2569 scope.go:117] "RemoveContainer" containerID="389086bb7301916abad1a385f6902b4651b51327fb2fdea83b2adbc996453c30"
Jul 17 21:55:57.190981 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:57.190960    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2"]
Jul 17 21:55:57.193475 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:57.193443    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5g7l2"]
Jul 17 21:55:57.440707 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:57.440626    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:57.539277 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:57.539248    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:58.169757 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:58.169725    2569 generic.go:356] "Generic (PLEG): container finished" podID="555c8dcd-c3d3-449e-a73b-587e1fad896b" containerID="14fffbc9496c5ad276e775fa568eb456d536b7b2bdb51dd873f0716234d51b61" exitCode=6
Jul 17 21:55:58.170241 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:58.169786    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq" event={"ID":"555c8dcd-c3d3-449e-a73b-587e1fad896b","Type":"ContainerDied","Data":"14fffbc9496c5ad276e775fa568eb456d536b7b2bdb51dd873f0716234d51b61"}
Jul 17 21:55:58.170241 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:58.169831    2569 scope.go:117] "RemoveContainer" containerID="389086bb7301916abad1a385f6902b4651b51327fb2fdea83b2adbc996453c30"
Jul 17 21:55:58.170241 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:58.170071    2569 scope.go:117] "RemoveContainer" containerID="14fffbc9496c5ad276e775fa568eb456d536b7b2bdb51dd873f0716234d51b61"
Jul 17 21:55:58.170432 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:55:58.170333    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq_odh-ai-gateway-infra(555c8dcd-c3d3-449e-a73b-587e1fad896b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq" podUID="555c8dcd-c3d3-449e-a73b-587e1fad896b"
Jul 17 21:55:58.657331 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:58.657301    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74c748e2-fd92-4660-a253-eefeb9b45126" path="/var/lib/kubelet/pods/74c748e2-fd92-4660-a253-eefeb9b45126/volumes"
Jul 17 21:55:59.176904 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:55:59.176868    2569 scope.go:117] "RemoveContainer" containerID="14fffbc9496c5ad276e775fa568eb456d536b7b2bdb51dd873f0716234d51b61"
Jul 17 21:55:59.177275 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:55:59.177072    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq_odh-ai-gateway-infra(555c8dcd-c3d3-449e-a73b-587e1fad896b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq" podUID="555c8dcd-c3d3-449e-a73b-587e1fad896b"
Jul 17 21:56:05.418529 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.418492    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"]
Jul 17 21:56:05.418906 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.418869    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74c748e2-fd92-4660-a253-eefeb9b45126" containerName="revoke-keys"
Jul 17 21:56:05.418906 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.418880    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="74c748e2-fd92-4660-a253-eefeb9b45126" containerName="revoke-keys"
Jul 17 21:56:05.418906 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.418894    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74c748e2-fd92-4660-a253-eefeb9b45126" containerName="revoke-keys"
Jul 17 21:56:05.418906 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.418900    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="74c748e2-fd92-4660-a253-eefeb9b45126" containerName="revoke-keys"
Jul 17 21:56:05.419104 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.418972    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="74c748e2-fd92-4660-a253-eefeb9b45126" containerName="revoke-keys"
Jul 17 21:56:05.419104 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.418980    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="74c748e2-fd92-4660-a253-eefeb9b45126" containerName="revoke-keys"
Jul 17 21:56:05.423765 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.423742    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:05.425765 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.425741    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ffd161\"/\"kube-root-ca.crt\""
Jul 17 21:56:05.425888 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.425806    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ffd161\"/\"openshift-service-ca.crt\""
Jul 17 21:56:05.426292 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.426274    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ffd161\"/\"test-model-ffd161-kserve-self-signed-certs\""
Jul 17 21:56:05.426373 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.426303    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ffd161\"/\"default-dockercfg-7rc4d\""
Jul 17 21:56:05.433691 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.433667    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"]
Jul 17 21:56:05.513505 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.513458    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlfxn\" (UniqueName: \"kubernetes.io/projected/0d3653f0-977e-42ff-9632-5edab244ce05-kube-api-access-mlfxn\") pod \"test-model-ffd161-kserve-69987cf49c-tfw7t\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") " pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:05.513701 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.513552    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-dshm\") pod \"test-model-ffd161-kserve-69987cf49c-tfw7t\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") " pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:05.513701 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.513602    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-model-cache\") pod \"test-model-ffd161-kserve-69987cf49c-tfw7t\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") " pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:05.513866 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.513737    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-home\") pod \"test-model-ffd161-kserve-69987cf49c-tfw7t\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") " pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:05.513866 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.513779    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-tmp-dir\") pod \"test-model-ffd161-kserve-69987cf49c-tfw7t\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") " pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:05.513866 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.513815    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3653f0-977e-42ff-9632-5edab244ce05-tls-certs\") pod \"test-model-ffd161-kserve-69987cf49c-tfw7t\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") " pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:05.518388 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.518364    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:56:05.614641 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.614609    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-home\") pod \"test-model-ffd161-kserve-69987cf49c-tfw7t\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") " pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:05.614802 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.614650    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-tmp-dir\") pod \"test-model-ffd161-kserve-69987cf49c-tfw7t\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") " pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:05.614802 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.614674    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3653f0-977e-42ff-9632-5edab244ce05-tls-certs\") pod \"test-model-ffd161-kserve-69987cf49c-tfw7t\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") " pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:05.614802 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.614697    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlfxn\" (UniqueName: \"kubernetes.io/projected/0d3653f0-977e-42ff-9632-5edab244ce05-kube-api-access-mlfxn\") pod \"test-model-ffd161-kserve-69987cf49c-tfw7t\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") " pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:05.614802 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.614720    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-dshm\") pod \"test-model-ffd161-kserve-69987cf49c-tfw7t\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") " pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:05.614802 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.614738    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-model-cache\") pod \"test-model-ffd161-kserve-69987cf49c-tfw7t\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") " pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:05.615104 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.615083    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-home\") pod \"test-model-ffd161-kserve-69987cf49c-tfw7t\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") " pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:05.615179 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.615101    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-tmp-dir\") pod \"test-model-ffd161-kserve-69987cf49c-tfw7t\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") " pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:05.615179 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.615155    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-model-cache\") pod \"test-model-ffd161-kserve-69987cf49c-tfw7t\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") " pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:05.616927 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.616906    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-dshm\") pod \"test-model-ffd161-kserve-69987cf49c-tfw7t\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") " pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:05.617264 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.617248    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3653f0-977e-42ff-9632-5edab244ce05-tls-certs\") pod \"test-model-ffd161-kserve-69987cf49c-tfw7t\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") " pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:05.621766 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.621745    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlfxn\" (UniqueName: \"kubernetes.io/projected/0d3653f0-977e-42ff-9632-5edab244ce05-kube-api-access-mlfxn\") pod \"test-model-ffd161-kserve-69987cf49c-tfw7t\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") " pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:05.733843 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.733758    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:05.869112 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.869086    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"]
Jul 17 21:56:05.870303 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:05.870266    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:56:05.871845 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:56:05.871815    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d3653f0_977e_42ff_9632_5edab244ce05.slice/crio-8a7e4466114069274b737d7071ca5c0d30e5ba4d8859e78b80d71e6fb58739a9 WatchSource:0}: Error finding container 8a7e4466114069274b737d7071ca5c0d30e5ba4d8859e78b80d71e6fb58739a9: Status 404 returned error can't find the container with id 8a7e4466114069274b737d7071ca5c0d30e5ba4d8859e78b80d71e6fb58739a9
Jul 17 21:56:06.219084 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:06.219047    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t" event={"ID":"0d3653f0-977e-42ff-9632-5edab244ce05","Type":"ContainerStarted","Data":"5d4101c9ccaf84689204b2898e5b829921092d64de5a3769dfe82b31e814a7e3"}
Jul 17 21:56:06.219084 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:06.219094    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t" event={"ID":"0d3653f0-977e-42ff-9632-5edab244ce05","Type":"ContainerStarted","Data":"8a7e4466114069274b737d7071ca5c0d30e5ba4d8859e78b80d71e6fb58739a9"}
Jul 17 21:56:06.236163 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:06.236114    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t" podStartSLOduration=1.236095898 podStartE2EDuration="1.236095898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:56:06.235446887 +0000 UTC m=+2879.995013581" watchObservedRunningTime="2026-07-17 21:56:06.236095898 +0000 UTC m=+2879.995662592"
Jul 17 21:56:07.226750 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:07.226713    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:56:07.326499 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:07.326465    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:56:07.650790 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:07.650763    2569 scope.go:117] "RemoveContainer" containerID="61243ba0c5e609f6f84f4b7fccf16f968917913be31ea93d11f5b05b3d81e315"
Jul 17 21:56:08.233496 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:08.233416    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h" event={"ID":"e380cc23-429a-47c6-8859-f829ccc9ba1e","Type":"ContainerStarted","Data":"1363b97276d54671323ac311134f3449f06e8676f081de0b5c7faabd9c9b9e79"}
Jul 17 21:56:08.251507 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:08.251459    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h" podStartSLOduration=19.251444474 podStartE2EDuration="19.251444474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:56:08.248484377 +0000 UTC m=+2882.008051071" watchObservedRunningTime="2026-07-17 21:56:08.251444474 +0000 UTC m=+2882.011011166"
Jul 17 21:56:09.217490 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:09.217452    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:56:09.240991 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:09.240957    2569 generic.go:356] "Generic (PLEG): container finished" podID="e380cc23-429a-47c6-8859-f829ccc9ba1e" containerID="1363b97276d54671323ac311134f3449f06e8676f081de0b5c7faabd9c9b9e79" exitCode=6
Jul 17 21:56:09.241440 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:09.241011    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h" event={"ID":"e380cc23-429a-47c6-8859-f829ccc9ba1e","Type":"ContainerDied","Data":"1363b97276d54671323ac311134f3449f06e8676f081de0b5c7faabd9c9b9e79"}
Jul 17 21:56:09.241440 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:09.241079    2569 scope.go:117] "RemoveContainer" containerID="61243ba0c5e609f6f84f4b7fccf16f968917913be31ea93d11f5b05b3d81e315"
Jul 17 21:56:09.241440 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:09.241333    2569 scope.go:117] "RemoveContainer" containerID="1363b97276d54671323ac311134f3449f06e8676f081de0b5c7faabd9c9b9e79"
Jul 17 21:56:09.241623 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:56:09.241560    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h_odh-ai-gateway-infra(e380cc23-429a-47c6-8859-f829ccc9ba1e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h" podUID="e380cc23-429a-47c6-8859-f829ccc9ba1e"
Jul 17 21:56:09.264005 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:09.263974    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h"]
Jul 17 21:56:10.391517 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:10.391491    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h"
Jul 17 21:56:10.463955 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:10.463927    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e380cc23-429a-47c6-8859-f829ccc9ba1e-maas-api-service-ca\") pod \"e380cc23-429a-47c6-8859-f829ccc9ba1e\" (UID: \"e380cc23-429a-47c6-8859-f829ccc9ba1e\") "
Jul 17 21:56:10.464288 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:10.464267    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e380cc23-429a-47c6-8859-f829ccc9ba1e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e380cc23-429a-47c6-8859-f829ccc9ba1e" (UID: "e380cc23-429a-47c6-8859-f829ccc9ba1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:56:10.565458 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:10.565371    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e380cc23-429a-47c6-8859-f829ccc9ba1e-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:56:11.255531 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:11.255494    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h" event={"ID":"e380cc23-429a-47c6-8859-f829ccc9ba1e","Type":"ContainerDied","Data":"9215fd662db7e65e705ae614abf1b51480ade2aca5ef3804dfd14bd7e1e55c40"}
Jul 17 21:56:11.255531 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:11.255520    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h"
Jul 17 21:56:11.255786 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:11.255569    2569 scope.go:117] "RemoveContainer" containerID="1363b97276d54671323ac311134f3449f06e8676f081de0b5c7faabd9c9b9e79"
Jul 17 21:56:11.271697 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:11.271659    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h"]
Jul 17 21:56:11.276946 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:11.276923    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5277h"]
Jul 17 21:56:12.650684 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:12.650650    2569 scope.go:117] "RemoveContainer" containerID="14fffbc9496c5ad276e775fa568eb456d536b7b2bdb51dd873f0716234d51b61"
Jul 17 21:56:12.657304 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:12.657274    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e380cc23-429a-47c6-8859-f829ccc9ba1e" path="/var/lib/kubelet/pods/e380cc23-429a-47c6-8859-f829ccc9ba1e/volumes"
Jul 17 21:56:13.271334 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:13.271304    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq" event={"ID":"555c8dcd-c3d3-449e-a73b-587e1fad896b","Type":"ContainerStarted","Data":"ba0e8b3136610d982caf62dcc84f378256405883f6ca6a4a02000c99162138d1"}
Jul 17 21:56:14.280063 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:14.280026    2569 generic.go:356] "Generic (PLEG): container finished" podID="555c8dcd-c3d3-449e-a73b-587e1fad896b" containerID="ba0e8b3136610d982caf62dcc84f378256405883f6ca6a4a02000c99162138d1" exitCode=6
Jul 17 21:56:14.280476 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:14.280096    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq" event={"ID":"555c8dcd-c3d3-449e-a73b-587e1fad896b","Type":"ContainerDied","Data":"ba0e8b3136610d982caf62dcc84f378256405883f6ca6a4a02000c99162138d1"}
Jul 17 21:56:14.280476 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:14.280156    2569 scope.go:117] "RemoveContainer" containerID="14fffbc9496c5ad276e775fa568eb456d536b7b2bdb51dd873f0716234d51b61"
Jul 17 21:56:14.280589 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:14.280484    2569 scope.go:117] "RemoveContainer" containerID="ba0e8b3136610d982caf62dcc84f378256405883f6ca6a4a02000c99162138d1"
Jul 17 21:56:14.280739 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:56:14.280721    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq_odh-ai-gateway-infra(555c8dcd-c3d3-449e-a73b-587e1fad896b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq" podUID="555c8dcd-c3d3-449e-a73b-587e1fad896b"
Jul 17 21:56:14.296983 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:14.296960    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq"]
Jul 17 21:56:15.463372 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:15.463347    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq"
Jul 17 21:56:15.501087 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:15.501055    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/555c8dcd-c3d3-449e-a73b-587e1fad896b-maas-api-service-ca\") pod \"555c8dcd-c3d3-449e-a73b-587e1fad896b\" (UID: \"555c8dcd-c3d3-449e-a73b-587e1fad896b\") "
Jul 17 21:56:15.501381 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:15.501357    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555c8dcd-c3d3-449e-a73b-587e1fad896b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "555c8dcd-c3d3-449e-a73b-587e1fad896b" (UID: "555c8dcd-c3d3-449e-a73b-587e1fad896b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:56:15.602110 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:15.602065    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/555c8dcd-c3d3-449e-a73b-587e1fad896b-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:56:15.734254 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:15.734221    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:15.734437 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:15.734267    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:15.750126 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:15.750093    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:16.295824 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:16.295794    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq"
Jul 17 21:56:16.296008 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:16.295831    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq" event={"ID":"555c8dcd-c3d3-449e-a73b-587e1fad896b","Type":"ContainerDied","Data":"c832b29ee9dd272d1480d1268afda45cf21099ce04f6ec0e05f6da02b2d3952d"}
Jul 17 21:56:16.296008 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:16.295891    2569 scope.go:117] "RemoveContainer" containerID="ba0e8b3136610d982caf62dcc84f378256405883f6ca6a4a02000c99162138d1"
Jul 17 21:56:16.310196 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:16.310175    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:56:16.319332 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:16.319305    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq"]
Jul 17 21:56:16.328386 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:16.328355    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-gtqtq"]
Jul 17 21:56:16.656486 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:16.656454    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="555c8dcd-c3d3-449e-a73b-587e1fad896b" path="/var/lib/kubelet/pods/555c8dcd-c3d3-449e-a73b-587e1fad896b/volumes"
Jul 17 21:56:20.529449 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:20.529395    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:56:27.706673 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.706643    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn"]
Jul 17 21:56:27.707056 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.707042    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="555c8dcd-c3d3-449e-a73b-587e1fad896b" containerName="revoke-keys"
Jul 17 21:56:27.707099 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.707057    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="555c8dcd-c3d3-449e-a73b-587e1fad896b" containerName="revoke-keys"
Jul 17 21:56:27.707099 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.707074    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e380cc23-429a-47c6-8859-f829ccc9ba1e" containerName="revoke-keys"
Jul 17 21:56:27.707099 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.707080    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e380cc23-429a-47c6-8859-f829ccc9ba1e" containerName="revoke-keys"
Jul 17 21:56:27.707099 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.707087    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74c748e2-fd92-4660-a253-eefeb9b45126" containerName="revoke-keys"
Jul 17 21:56:27.707099 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.707092    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="74c748e2-fd92-4660-a253-eefeb9b45126" containerName="revoke-keys"
Jul 17 21:56:27.707287 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.707103    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e380cc23-429a-47c6-8859-f829ccc9ba1e" containerName="revoke-keys"
Jul 17 21:56:27.707287 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.707111    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e380cc23-429a-47c6-8859-f829ccc9ba1e" containerName="revoke-keys"
Jul 17 21:56:27.707287 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.707120    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="555c8dcd-c3d3-449e-a73b-587e1fad896b" containerName="revoke-keys"
Jul 17 21:56:27.707287 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.707126    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="555c8dcd-c3d3-449e-a73b-587e1fad896b" containerName="revoke-keys"
Jul 17 21:56:27.707287 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.707216    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e380cc23-429a-47c6-8859-f829ccc9ba1e" containerName="revoke-keys"
Jul 17 21:56:27.707287 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.707225    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="555c8dcd-c3d3-449e-a73b-587e1fad896b" containerName="revoke-keys"
Jul 17 21:56:27.707287 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.707231    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e380cc23-429a-47c6-8859-f829ccc9ba1e" containerName="revoke-keys"
Jul 17 21:56:27.707287 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.707238    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="555c8dcd-c3d3-449e-a73b-587e1fad896b" containerName="revoke-keys"
Jul 17 21:56:27.707287 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.707245    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e380cc23-429a-47c6-8859-f829ccc9ba1e" containerName="revoke-keys"
Jul 17 21:56:27.707287 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.707252    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="74c748e2-fd92-4660-a253-eefeb9b45126" containerName="revoke-keys"
Jul 17 21:56:27.711848 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.711831    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn"
Jul 17 21:56:27.714480 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.714454    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:56:27.714611 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.714485    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:56:27.717289 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.717263    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn"]
Jul 17 21:56:27.810287 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.810259    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff6b080-25a8-4156-aea3-13b54f068358-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn\" (UID: \"aff6b080-25a8-4156-aea3-13b54f068358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn"
Jul 17 21:56:27.911188 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.911155    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff6b080-25a8-4156-aea3-13b54f068358-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn\" (UID: \"aff6b080-25a8-4156-aea3-13b54f068358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn"
Jul 17 21:56:27.911805 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:27.911785    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff6b080-25a8-4156-aea3-13b54f068358-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn\" (UID: \"aff6b080-25a8-4156-aea3-13b54f068358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn"
Jul 17 21:56:28.024322 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:28.024242    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn"
Jul 17 21:56:28.160544 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:28.160509    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn"]
Jul 17 21:56:28.161516 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:56:28.161489    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff6b080_25a8_4156_aea3_13b54f068358.slice/crio-3310533813e5e6549e1de8e86ffcfebaf565d1e184a7db50b9ab9a544390960d WatchSource:0}: Error finding container 3310533813e5e6549e1de8e86ffcfebaf565d1e184a7db50b9ab9a544390960d: Status 404 returned error can't find the container with id 3310533813e5e6549e1de8e86ffcfebaf565d1e184a7db50b9ab9a544390960d
Jul 17 21:56:28.366895 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:28.366861    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn" event={"ID":"aff6b080-25a8-4156-aea3-13b54f068358","Type":"ContainerStarted","Data":"3310533813e5e6549e1de8e86ffcfebaf565d1e184a7db50b9ab9a544390960d"}
Jul 17 21:56:29.374786 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:29.374748    2569 generic.go:356] "Generic (PLEG): container finished" podID="aff6b080-25a8-4156-aea3-13b54f068358" containerID="72edd3db0858b3e6102c8f123f045a48a830d094610cccd545c1835c80ffdffe" exitCode=6
Jul 17 21:56:29.375192 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:29.374831    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn" event={"ID":"aff6b080-25a8-4156-aea3-13b54f068358","Type":"ContainerDied","Data":"72edd3db0858b3e6102c8f123f045a48a830d094610cccd545c1835c80ffdffe"}
Jul 17 21:56:29.375192 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:29.375006    2569 scope.go:117] "RemoveContainer" containerID="72edd3db0858b3e6102c8f123f045a48a830d094610cccd545c1835c80ffdffe"
Jul 17 21:56:30.386837 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:30.386802    2569 generic.go:356] "Generic (PLEG): container finished" podID="aff6b080-25a8-4156-aea3-13b54f068358" containerID="a77340af8c505ebf1d28677012dd20bb5f6e26398fcf85d862c66d376e2a6bcb" exitCode=6
Jul 17 21:56:30.387315 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:30.386881    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn" event={"ID":"aff6b080-25a8-4156-aea3-13b54f068358","Type":"ContainerDied","Data":"a77340af8c505ebf1d28677012dd20bb5f6e26398fcf85d862c66d376e2a6bcb"}
Jul 17 21:56:30.387315 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:30.386949    2569 scope.go:117] "RemoveContainer" containerID="72edd3db0858b3e6102c8f123f045a48a830d094610cccd545c1835c80ffdffe"
Jul 17 21:56:30.387315 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:30.387118    2569 scope.go:117] "RemoveContainer" containerID="a77340af8c505ebf1d28677012dd20bb5f6e26398fcf85d862c66d376e2a6bcb"
Jul 17 21:56:30.387506 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:56:30.387326    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn_odh-ai-gateway-infra(aff6b080-25a8-4156-aea3-13b54f068358)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn" podUID="aff6b080-25a8-4156-aea3-13b54f068358"
Jul 17 21:56:30.761090 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:30.761006    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266"]
Jul 17 21:56:30.761511 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:30.761492    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="555c8dcd-c3d3-449e-a73b-587e1fad896b" containerName="revoke-keys"
Jul 17 21:56:30.761620 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:30.761513    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="555c8dcd-c3d3-449e-a73b-587e1fad896b" containerName="revoke-keys"
Jul 17 21:56:30.761620 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:30.761541    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e380cc23-429a-47c6-8859-f829ccc9ba1e" containerName="revoke-keys"
Jul 17 21:56:30.761620 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:30.761551    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e380cc23-429a-47c6-8859-f829ccc9ba1e" containerName="revoke-keys"
Jul 17 21:56:30.761775 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:30.761664    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="555c8dcd-c3d3-449e-a73b-587e1fad896b" containerName="revoke-keys"
Jul 17 21:56:30.765885 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:30.765865    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266"
Jul 17 21:56:30.773319 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:30.773290    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266"]
Jul 17 21:56:30.839884 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:30.839856    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00facfa3-8bea-4459-989e-ea434fe6932e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-vv266\" (UID: \"00facfa3-8bea-4459-989e-ea434fe6932e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266"
Jul 17 21:56:30.940711 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:30.940671    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00facfa3-8bea-4459-989e-ea434fe6932e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-vv266\" (UID: \"00facfa3-8bea-4459-989e-ea434fe6932e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266"
Jul 17 21:56:30.941298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:30.941277    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00facfa3-8bea-4459-989e-ea434fe6932e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-vv266\" (UID: \"00facfa3-8bea-4459-989e-ea434fe6932e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266"
Jul 17 21:56:31.078852 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:31.078763    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266"
Jul 17 21:56:31.215639 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:31.215610    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266"]
Jul 17 21:56:31.216255 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:56:31.216228    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00facfa3_8bea_4459_989e_ea434fe6932e.slice/crio-c1a960980aa458c0fcfe93921086b20d295d63b5c0dd77aba6621a918916f051 WatchSource:0}: Error finding container c1a960980aa458c0fcfe93921086b20d295d63b5c0dd77aba6621a918916f051: Status 404 returned error can't find the container with id c1a960980aa458c0fcfe93921086b20d295d63b5c0dd77aba6621a918916f051
Jul 17 21:56:31.395186 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:31.395153    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266" event={"ID":"00facfa3-8bea-4459-989e-ea434fe6932e","Type":"ContainerStarted","Data":"c1a960980aa458c0fcfe93921086b20d295d63b5c0dd77aba6621a918916f051"}
Jul 17 21:56:31.397155 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:31.397136    2569 scope.go:117] "RemoveContainer" containerID="a77340af8c505ebf1d28677012dd20bb5f6e26398fcf85d862c66d376e2a6bcb"
Jul 17 21:56:31.397376 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:56:31.397355    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn_odh-ai-gateway-infra(aff6b080-25a8-4156-aea3-13b54f068358)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn" podUID="aff6b080-25a8-4156-aea3-13b54f068358"
Jul 17 21:56:32.405075 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:32.405038    2569 generic.go:356] "Generic (PLEG): container finished" podID="00facfa3-8bea-4459-989e-ea434fe6932e" containerID="5c2147a25c50065abf14897217ebc7305eeb0c14daa358c50f9e1ffbc20f3c35" exitCode=6
Jul 17 21:56:32.405498 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:32.405094    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266" event={"ID":"00facfa3-8bea-4459-989e-ea434fe6932e","Type":"ContainerDied","Data":"5c2147a25c50065abf14897217ebc7305eeb0c14daa358c50f9e1ffbc20f3c35"}
Jul 17 21:56:32.405498 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:32.405305    2569 scope.go:117] "RemoveContainer" containerID="5c2147a25c50065abf14897217ebc7305eeb0c14daa358c50f9e1ffbc20f3c35"
Jul 17 21:56:33.412069 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:33.412028    2569 generic.go:356] "Generic (PLEG): container finished" podID="00facfa3-8bea-4459-989e-ea434fe6932e" containerID="3be948684d96da94b59abd615e528cb3c71eb59aa0cfcdd39f8a10cdf592bd20" exitCode=6
Jul 17 21:56:33.412519 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:33.412104    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266" event={"ID":"00facfa3-8bea-4459-989e-ea434fe6932e","Type":"ContainerDied","Data":"3be948684d96da94b59abd615e528cb3c71eb59aa0cfcdd39f8a10cdf592bd20"}
Jul 17 21:56:33.412519 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:33.412189    2569 scope.go:117] "RemoveContainer" containerID="5c2147a25c50065abf14897217ebc7305eeb0c14daa358c50f9e1ffbc20f3c35"
Jul 17 21:56:33.412519 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:33.412372    2569 scope.go:117] "RemoveContainer" containerID="3be948684d96da94b59abd615e528cb3c71eb59aa0cfcdd39f8a10cdf592bd20"
Jul 17 21:56:33.412642 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:56:33.412615    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-vv266_odh-ai-gateway-infra(00facfa3-8bea-4459-989e-ea434fe6932e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266" podUID="00facfa3-8bea-4459-989e-ea434fe6932e"
Jul 17 21:56:34.420540 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:34.420510    2569 scope.go:117] "RemoveContainer" containerID="3be948684d96da94b59abd615e528cb3c71eb59aa0cfcdd39f8a10cdf592bd20"
Jul 17 21:56:34.420895 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:56:34.420696    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-vv266_odh-ai-gateway-infra(00facfa3-8bea-4459-989e-ea434fe6932e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266" podUID="00facfa3-8bea-4459-989e-ea434fe6932e"
Jul 17 21:56:45.323680 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.323625    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"]
Jul 17 21:56:45.328858 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.328835    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:45.332481 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.332455    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-786043\"/\"test-model-786043-kserve-self-signed-certs\""
Jul 17 21:56:45.332616 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.332486    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-786043\"/\"default-dockercfg-rr94f\""
Jul 17 21:56:45.332616 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.332579    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-786043\"/\"kube-root-ca.crt\""
Jul 17 21:56:45.332868 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.332848    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-786043\"/\"openshift-service-ca.crt\""
Jul 17 21:56:45.340677 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.340653    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"]
Jul 17 21:56:45.359037 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.359001    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:56:45.466176 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.466138    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-model-cache\") pod \"test-model-786043-kserve-5fd8777dd9-cx7rp\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") " pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:45.466176 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.466181    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-home\") pod \"test-model-786043-kserve-5fd8777dd9-cx7rp\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") " pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:45.466493 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.466210    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-tmp-dir\") pod \"test-model-786043-kserve-5fd8777dd9-cx7rp\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") " pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:45.466493 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.466312    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-dshm\") pod \"test-model-786043-kserve-5fd8777dd9-cx7rp\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") " pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:45.466493 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.466356    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvs8q\" (UniqueName: \"kubernetes.io/projected/89df7841-881a-4b99-9268-f3aa220816d4-kube-api-access-jvs8q\") pod \"test-model-786043-kserve-5fd8777dd9-cx7rp\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") " pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:45.466493 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.466388    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89df7841-881a-4b99-9268-f3aa220816d4-tls-certs\") pod \"test-model-786043-kserve-5fd8777dd9-cx7rp\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") " pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:45.567374 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.567331    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-home\") pod \"test-model-786043-kserve-5fd8777dd9-cx7rp\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") " pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:45.567591 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.567388    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-tmp-dir\") pod \"test-model-786043-kserve-5fd8777dd9-cx7rp\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") " pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:45.567591 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.567491    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-dshm\") pod \"test-model-786043-kserve-5fd8777dd9-cx7rp\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") " pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:45.567591 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.567522    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvs8q\" (UniqueName: \"kubernetes.io/projected/89df7841-881a-4b99-9268-f3aa220816d4-kube-api-access-jvs8q\") pod \"test-model-786043-kserve-5fd8777dd9-cx7rp\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") " pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:45.567591 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.567561    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89df7841-881a-4b99-9268-f3aa220816d4-tls-certs\") pod \"test-model-786043-kserve-5fd8777dd9-cx7rp\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") " pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:45.567842 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.567637    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-model-cache\") pod \"test-model-786043-kserve-5fd8777dd9-cx7rp\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") " pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:45.567842 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.567818    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-home\") pod \"test-model-786043-kserve-5fd8777dd9-cx7rp\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") " pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:45.567958 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.567869    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-tmp-dir\") pod \"test-model-786043-kserve-5fd8777dd9-cx7rp\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") " pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:45.568019 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.567987    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-model-cache\") pod \"test-model-786043-kserve-5fd8777dd9-cx7rp\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") " pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:45.569858 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.569831    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-dshm\") pod \"test-model-786043-kserve-5fd8777dd9-cx7rp\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") " pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:45.570100 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.570083    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89df7841-881a-4b99-9268-f3aa220816d4-tls-certs\") pod \"test-model-786043-kserve-5fd8777dd9-cx7rp\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") " pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:45.574970 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.574913    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvs8q\" (UniqueName: \"kubernetes.io/projected/89df7841-881a-4b99-9268-f3aa220816d4-kube-api-access-jvs8q\") pod \"test-model-786043-kserve-5fd8777dd9-cx7rp\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") " pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:45.639773 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.639716    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:45.650232 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.650211    2569 scope.go:117] "RemoveContainer" containerID="a77340af8c505ebf1d28677012dd20bb5f6e26398fcf85d862c66d376e2a6bcb"
Jul 17 21:56:45.791352 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:45.791322    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"]
Jul 17 21:56:45.792604 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:56:45.792575    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89df7841_881a_4b99_9268_f3aa220816d4.slice/crio-07867618aee9798edb1722c597aec4c40ee3600cde706d189a87563da5520ac0 WatchSource:0}: Error finding container 07867618aee9798edb1722c597aec4c40ee3600cde706d189a87563da5520ac0: Status 404 returned error can't find the container with id 07867618aee9798edb1722c597aec4c40ee3600cde706d189a87563da5520ac0
Jul 17 21:56:46.493595 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:46.493553    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp" event={"ID":"89df7841-881a-4b99-9268-f3aa220816d4","Type":"ContainerStarted","Data":"118b2048f948a6017a7c66226b7c39b084666a5888316c35009bf213564ccdff"}
Jul 17 21:56:46.493983 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:46.493606    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp" event={"ID":"89df7841-881a-4b99-9268-f3aa220816d4","Type":"ContainerStarted","Data":"07867618aee9798edb1722c597aec4c40ee3600cde706d189a87563da5520ac0"}
Jul 17 21:56:46.495389 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:46.495366    2569 generic.go:356] "Generic (PLEG): container finished" podID="aff6b080-25a8-4156-aea3-13b54f068358" containerID="6e278a2ef431ba57d50c4f7b6b325625cc6d56d35ef7f69c7b13e71a1d7d4ab6" exitCode=6
Jul 17 21:56:46.495520 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:46.495431    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn" event={"ID":"aff6b080-25a8-4156-aea3-13b54f068358","Type":"ContainerDied","Data":"6e278a2ef431ba57d50c4f7b6b325625cc6d56d35ef7f69c7b13e71a1d7d4ab6"}
Jul 17 21:56:46.495520 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:46.495510    2569 scope.go:117] "RemoveContainer" containerID="a77340af8c505ebf1d28677012dd20bb5f6e26398fcf85d862c66d376e2a6bcb"
Jul 17 21:56:46.495744 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:46.495730    2569 scope.go:117] "RemoveContainer" containerID="6e278a2ef431ba57d50c4f7b6b325625cc6d56d35ef7f69c7b13e71a1d7d4ab6"
Jul 17 21:56:46.495950 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:56:46.495934    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn_odh-ai-gateway-infra(aff6b080-25a8-4156-aea3-13b54f068358)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn" podUID="aff6b080-25a8-4156-aea3-13b54f068358"
Jul 17 21:56:46.509092 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:46.509047    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp" podStartSLOduration=1.5090335750000001 podStartE2EDuration="1.509033575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:56:46.508462128 +0000 UTC m=+2920.268028843" watchObservedRunningTime="2026-07-17 21:56:46.509033575 +0000 UTC m=+2920.268600267"
Jul 17 21:56:46.653804 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:46.653770    2569 scope.go:117] "RemoveContainer" containerID="3be948684d96da94b59abd615e528cb3c71eb59aa0cfcdd39f8a10cdf592bd20"
Jul 17 21:56:47.504515 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:47.504474    2569 generic.go:356] "Generic (PLEG): container finished" podID="00facfa3-8bea-4459-989e-ea434fe6932e" containerID="14e4ae78f680a5db25336b76cc518a340c885cd092dbb4e3faad46c44d3898d6" exitCode=6
Jul 17 21:56:47.504975 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:47.504547    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266" event={"ID":"00facfa3-8bea-4459-989e-ea434fe6932e","Type":"ContainerDied","Data":"14e4ae78f680a5db25336b76cc518a340c885cd092dbb4e3faad46c44d3898d6"}
Jul 17 21:56:47.504975 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:47.504603    2569 scope.go:117] "RemoveContainer" containerID="3be948684d96da94b59abd615e528cb3c71eb59aa0cfcdd39f8a10cdf592bd20"
Jul 17 21:56:47.504975 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:47.504930    2569 scope.go:117] "RemoveContainer" containerID="14e4ae78f680a5db25336b76cc518a340c885cd092dbb4e3faad46c44d3898d6"
Jul 17 21:56:47.505145 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:56:47.505120    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-vv266_odh-ai-gateway-infra(00facfa3-8bea-4459-989e-ea434fe6932e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266" podUID="00facfa3-8bea-4459-989e-ea434fe6932e"
Jul 17 21:56:47.532100 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:47.532076    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn"]
Jul 17 21:56:47.672364 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:47.672337    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn"
Jul 17 21:56:47.787232 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:47.787147    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff6b080-25a8-4156-aea3-13b54f068358-maas-api-service-ca\") pod \"aff6b080-25a8-4156-aea3-13b54f068358\" (UID: \"aff6b080-25a8-4156-aea3-13b54f068358\") "
Jul 17 21:56:47.787535 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:47.787515    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff6b080-25a8-4156-aea3-13b54f068358-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aff6b080-25a8-4156-aea3-13b54f068358" (UID: "aff6b080-25a8-4156-aea3-13b54f068358"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:56:47.888139 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:47.888108    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff6b080-25a8-4156-aea3-13b54f068358-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:56:48.511182 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:48.511151    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn"
Jul 17 21:56:48.511641 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:48.511193    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn" event={"ID":"aff6b080-25a8-4156-aea3-13b54f068358","Type":"ContainerDied","Data":"3310533813e5e6549e1de8e86ffcfebaf565d1e184a7db50b9ab9a544390960d"}
Jul 17 21:56:48.511641 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:48.511256    2569 scope.go:117] "RemoveContainer" containerID="6e278a2ef431ba57d50c4f7b6b325625cc6d56d35ef7f69c7b13e71a1d7d4ab6"
Jul 17 21:56:48.531226 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:48.531201    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266"]
Jul 17 21:56:48.536068 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:48.536042    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn"]
Jul 17 21:56:48.541692 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:48.541669    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8cpvn"]
Jul 17 21:56:48.657302 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:48.657273    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aff6b080-25a8-4156-aea3-13b54f068358" path="/var/lib/kubelet/pods/aff6b080-25a8-4156-aea3-13b54f068358/volumes"
Jul 17 21:56:48.674066 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:48.674044    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266"
Jul 17 21:56:48.797230 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:48.797152    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00facfa3-8bea-4459-989e-ea434fe6932e-maas-api-service-ca\") pod \"00facfa3-8bea-4459-989e-ea434fe6932e\" (UID: \"00facfa3-8bea-4459-989e-ea434fe6932e\") "
Jul 17 21:56:48.797530 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:48.797509    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00facfa3-8bea-4459-989e-ea434fe6932e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "00facfa3-8bea-4459-989e-ea434fe6932e" (UID: "00facfa3-8bea-4459-989e-ea434fe6932e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:56:48.898032 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:48.897989    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00facfa3-8bea-4459-989e-ea434fe6932e-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:56:49.520569 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.520537    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266"
Jul 17 21:56:49.521058 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.520565    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266" event={"ID":"00facfa3-8bea-4459-989e-ea434fe6932e","Type":"ContainerDied","Data":"c1a960980aa458c0fcfe93921086b20d295d63b5c0dd77aba6621a918916f051"}
Jul 17 21:56:49.521058 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.520665    2569 scope.go:117] "RemoveContainer" containerID="14e4ae78f680a5db25336b76cc518a340c885cd092dbb4e3faad46c44d3898d6"
Jul 17 21:56:49.574154 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.574126    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266"]
Jul 17 21:56:49.576458 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.576437    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vv266"]
Jul 17 21:56:49.633241 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.633209    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz"]
Jul 17 21:56:49.633597 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.633584    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff6b080-25a8-4156-aea3-13b54f068358" containerName="revoke-keys"
Jul 17 21:56:49.633638 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.633599    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff6b080-25a8-4156-aea3-13b54f068358" containerName="revoke-keys"
Jul 17 21:56:49.633638 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.633610    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00facfa3-8bea-4459-989e-ea434fe6932e" containerName="revoke-keys"
Jul 17 21:56:49.633638 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.633616    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="00facfa3-8bea-4459-989e-ea434fe6932e" containerName="revoke-keys"
Jul 17 21:56:49.633638 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.633623    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff6b080-25a8-4156-aea3-13b54f068358" containerName="revoke-keys"
Jul 17 21:56:49.633638 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.633630    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff6b080-25a8-4156-aea3-13b54f068358" containerName="revoke-keys"
Jul 17 21:56:49.633638 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.633636    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00facfa3-8bea-4459-989e-ea434fe6932e" containerName="revoke-keys"
Jul 17 21:56:49.633823 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.633641    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="00facfa3-8bea-4459-989e-ea434fe6932e" containerName="revoke-keys"
Jul 17 21:56:49.633823 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.633716    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="00facfa3-8bea-4459-989e-ea434fe6932e" containerName="revoke-keys"
Jul 17 21:56:49.633823 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.633723    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="aff6b080-25a8-4156-aea3-13b54f068358" containerName="revoke-keys"
Jul 17 21:56:49.633823 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.633728    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="00facfa3-8bea-4459-989e-ea434fe6932e" containerName="revoke-keys"
Jul 17 21:56:49.633823 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.633736    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="aff6b080-25a8-4156-aea3-13b54f068358" containerName="revoke-keys"
Jul 17 21:56:49.638004 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.637984    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz"
Jul 17 21:56:49.641072 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.641054    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:56:49.646438 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.646371    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:56:49.649655 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.649634    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz"]
Jul 17 21:56:49.705716 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.705684    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91f78a14-22f2-4e78-83e8-913dc57166b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz\" (UID: \"91f78a14-22f2-4e78-83e8-913dc57166b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz"
Jul 17 21:56:49.806800 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.806709    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91f78a14-22f2-4e78-83e8-913dc57166b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz\" (UID: \"91f78a14-22f2-4e78-83e8-913dc57166b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz"
Jul 17 21:56:49.807355 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.807336    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91f78a14-22f2-4e78-83e8-913dc57166b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz\" (UID: \"91f78a14-22f2-4e78-83e8-913dc57166b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz"
Jul 17 21:56:49.820088 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.820060    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:56:49.949522 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:49.949481    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz"
Jul 17 21:56:50.111665 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:50.111638    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz"]
Jul 17 21:56:50.113227 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:56:50.113196    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f78a14_22f2_4e78_83e8_913dc57166b6.slice/crio-e3df4da99f7be7209f16b172f2599c5485185de3062f69085d7c85a16f00d39f WatchSource:0}: Error finding container e3df4da99f7be7209f16b172f2599c5485185de3062f69085d7c85a16f00d39f: Status 404 returned error can't find the container with id e3df4da99f7be7209f16b172f2599c5485185de3062f69085d7c85a16f00d39f
Jul 17 21:56:50.529806 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:50.529760    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz" event={"ID":"91f78a14-22f2-4e78-83e8-913dc57166b6","Type":"ContainerStarted","Data":"53330046f35a950ed25a1804ae66ecd0b43513908ba4ce97658c0d4ed1b2a5be"}
Jul 17 21:56:50.530163 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:50.529825    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz" event={"ID":"91f78a14-22f2-4e78-83e8-913dc57166b6","Type":"ContainerStarted","Data":"e3df4da99f7be7209f16b172f2599c5485185de3062f69085d7c85a16f00d39f"}
Jul 17 21:56:50.545674 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:50.545619    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz" podStartSLOduration=1.5456043350000002 podStartE2EDuration="1.545604335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:56:50.54306174 +0000 UTC m=+2924.302628433" watchObservedRunningTime="2026-07-17 21:56:50.545604335 +0000 UTC m=+2924.305171027"
Jul 17 21:56:50.656377 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:50.656298    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00facfa3-8bea-4459-989e-ea434fe6932e" path="/var/lib/kubelet/pods/00facfa3-8bea-4459-989e-ea434fe6932e/volumes"
Jul 17 21:56:51.536834 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:51.536807    2569 generic.go:356] "Generic (PLEG): container finished" podID="91f78a14-22f2-4e78-83e8-913dc57166b6" containerID="53330046f35a950ed25a1804ae66ecd0b43513908ba4ce97658c0d4ed1b2a5be" exitCode=6
Jul 17 21:56:51.537232 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:51.536889    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz" event={"ID":"91f78a14-22f2-4e78-83e8-913dc57166b6","Type":"ContainerDied","Data":"53330046f35a950ed25a1804ae66ecd0b43513908ba4ce97658c0d4ed1b2a5be"}
Jul 17 21:56:51.537232 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:51.537065    2569 scope.go:117] "RemoveContainer" containerID="53330046f35a950ed25a1804ae66ecd0b43513908ba4ce97658c0d4ed1b2a5be"
Jul 17 21:56:52.544432 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:52.544385    2569 generic.go:356] "Generic (PLEG): container finished" podID="91f78a14-22f2-4e78-83e8-913dc57166b6" containerID="e890a7a2a46fe6655f8bdc4381938f617a933877976e42dfefdc24013dd68c9a" exitCode=6
Jul 17 21:56:52.544833 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:52.544473    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz" event={"ID":"91f78a14-22f2-4e78-83e8-913dc57166b6","Type":"ContainerDied","Data":"e890a7a2a46fe6655f8bdc4381938f617a933877976e42dfefdc24013dd68c9a"}
Jul 17 21:56:52.544833 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:52.544532    2569 scope.go:117] "RemoveContainer" containerID="53330046f35a950ed25a1804ae66ecd0b43513908ba4ce97658c0d4ed1b2a5be"
Jul 17 21:56:52.544833 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:52.544759    2569 scope.go:117] "RemoveContainer" containerID="e890a7a2a46fe6655f8bdc4381938f617a933877976e42dfefdc24013dd68c9a"
Jul 17 21:56:52.545045 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:56:52.545025    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz_odh-ai-gateway-infra(91f78a14-22f2-4e78-83e8-913dc57166b6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz" podUID="91f78a14-22f2-4e78-83e8-913dc57166b6"
Jul 17 21:56:53.551506 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:53.551477    2569 scope.go:117] "RemoveContainer" containerID="e890a7a2a46fe6655f8bdc4381938f617a933877976e42dfefdc24013dd68c9a"
Jul 17 21:56:53.551918 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:56:53.551663    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz_odh-ai-gateway-infra(91f78a14-22f2-4e78-83e8-913dc57166b6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz" podUID="91f78a14-22f2-4e78-83e8-913dc57166b6"
Jul 17 21:56:55.488236 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:55.488197    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7"]
Jul 17 21:56:55.488702 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:55.488686    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff6b080-25a8-4156-aea3-13b54f068358" containerName="revoke-keys"
Jul 17 21:56:55.488702 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:55.488703    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff6b080-25a8-4156-aea3-13b54f068358" containerName="revoke-keys"
Jul 17 21:56:55.488791 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:55.488713    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00facfa3-8bea-4459-989e-ea434fe6932e" containerName="revoke-keys"
Jul 17 21:56:55.488791 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:55.488721    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="00facfa3-8bea-4459-989e-ea434fe6932e" containerName="revoke-keys"
Jul 17 21:56:55.488875 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:55.488813    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="00facfa3-8bea-4459-989e-ea434fe6932e" containerName="revoke-keys"
Jul 17 21:56:55.488875 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:55.488826    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="aff6b080-25a8-4156-aea3-13b54f068358" containerName="revoke-keys"
Jul 17 21:56:55.493340 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:55.493322    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7"
Jul 17 21:56:55.498903 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:55.498877    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7"]
Jul 17 21:56:55.640628 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:55.640597    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:55.640801 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:55.640641    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:55.653089 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:55.653065    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:55.655922 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:55.655901    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37670757-bf7a-4566-82ac-7f2879a837c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7\" (UID: \"37670757-bf7a-4566-82ac-7f2879a837c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7"
Jul 17 21:56:55.757320 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:55.757247    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37670757-bf7a-4566-82ac-7f2879a837c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7\" (UID: \"37670757-bf7a-4566-82ac-7f2879a837c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7"
Jul 17 21:56:55.758094 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:55.758067    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37670757-bf7a-4566-82ac-7f2879a837c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7\" (UID: \"37670757-bf7a-4566-82ac-7f2879a837c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7"
Jul 17 21:56:55.806617 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:55.806581    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7"
Jul 17 21:56:55.941889 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:55.941862    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7"]
Jul 17 21:56:55.944035 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:56:55.944003    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37670757_bf7a_4566_82ac_7f2879a837c6.slice/crio-b0b987de2817da06e023eec0e0bbbcf620f328bf9265b63e3aac4aa5304f2a95 WatchSource:0}: Error finding container b0b987de2817da06e023eec0e0bbbcf620f328bf9265b63e3aac4aa5304f2a95: Status 404 returned error can't find the container with id b0b987de2817da06e023eec0e0bbbcf620f328bf9265b63e3aac4aa5304f2a95
Jul 17 21:56:56.573830 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:56.573794    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7" event={"ID":"37670757-bf7a-4566-82ac-7f2879a837c6","Type":"ContainerStarted","Data":"530558b0a2bcbe287ba2a26d5716d893f16ce676900ce8c522ce597d61e3360e"}
Jul 17 21:56:56.574202 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:56.573859    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7" event={"ID":"37670757-bf7a-4566-82ac-7f2879a837c6","Type":"ContainerStarted","Data":"b0b987de2817da06e023eec0e0bbbcf620f328bf9265b63e3aac4aa5304f2a95"}
Jul 17 21:56:56.585291 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:56.585269    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:56:56.588181 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:56.588141    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7" podStartSLOduration=1.588126222 podStartE2EDuration="1.588126222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:56:56.586309933 +0000 UTC m=+2930.345876627" watchObservedRunningTime="2026-07-17 21:56:56.588126222 +0000 UTC m=+2930.347692916"
Jul 17 21:56:57.580955 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:57.580920    2569 generic.go:356] "Generic (PLEG): container finished" podID="37670757-bf7a-4566-82ac-7f2879a837c6" containerID="530558b0a2bcbe287ba2a26d5716d893f16ce676900ce8c522ce597d61e3360e" exitCode=6
Jul 17 21:56:57.581442 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:57.581002    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7" event={"ID":"37670757-bf7a-4566-82ac-7f2879a837c6","Type":"ContainerDied","Data":"530558b0a2bcbe287ba2a26d5716d893f16ce676900ce8c522ce597d61e3360e"}
Jul 17 21:56:57.581442 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:57.581178    2569 scope.go:117] "RemoveContainer" containerID="530558b0a2bcbe287ba2a26d5716d893f16ce676900ce8c522ce597d61e3360e"
Jul 17 21:56:58.589344 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:58.589311    2569 generic.go:356] "Generic (PLEG): container finished" podID="37670757-bf7a-4566-82ac-7f2879a837c6" containerID="3bd17edca2539cdd990f32ad489af3eb22856bb540dbd7a36e8263f0ce30f149" exitCode=6
Jul 17 21:56:58.589825 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:58.589394    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7" event={"ID":"37670757-bf7a-4566-82ac-7f2879a837c6","Type":"ContainerDied","Data":"3bd17edca2539cdd990f32ad489af3eb22856bb540dbd7a36e8263f0ce30f149"}
Jul 17 21:56:58.589825 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:58.589480    2569 scope.go:117] "RemoveContainer" containerID="530558b0a2bcbe287ba2a26d5716d893f16ce676900ce8c522ce597d61e3360e"
Jul 17 21:56:58.589825 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:58.589657    2569 scope.go:117] "RemoveContainer" containerID="3bd17edca2539cdd990f32ad489af3eb22856bb540dbd7a36e8263f0ce30f149"
Jul 17 21:56:58.589977 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:56:58.589842    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7_odh-ai-gateway-infra(37670757-bf7a-4566-82ac-7f2879a837c6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7" podUID="37670757-bf7a-4566-82ac-7f2879a837c6"
Jul 17 21:56:59.597020 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:56:59.596984    2569 scope.go:117] "RemoveContainer" containerID="3bd17edca2539cdd990f32ad489af3eb22856bb540dbd7a36e8263f0ce30f149"
Jul 17 21:56:59.597425 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:56:59.597199    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7_odh-ai-gateway-infra(37670757-bf7a-4566-82ac-7f2879a837c6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7" podUID="37670757-bf7a-4566-82ac-7f2879a837c6"
Jul 17 21:57:07.167093 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:07.167054    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"]
Jul 17 21:57:07.173258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:07.172998    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:08.056809 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:08.056774    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"]
Jul 17 21:57:08.343103 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:08.343077    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:08.650233 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:08.650148    2569 scope.go:117] "RemoveContainer" containerID="e890a7a2a46fe6655f8bdc4381938f617a933877976e42dfefdc24013dd68c9a"
Jul 17 21:57:08.807514 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:08.807481    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:08.908449 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:08.908348    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:09.656008 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:09.655973    2569 generic.go:356] "Generic (PLEG): container finished" podID="91f78a14-22f2-4e78-83e8-913dc57166b6" containerID="9e22e5e31f60c8023b63236694bf52afe6d97eabe71fa404eb0abe3d77bbbb01" exitCode=6
Jul 17 21:57:09.656439 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:09.656046    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz" event={"ID":"91f78a14-22f2-4e78-83e8-913dc57166b6","Type":"ContainerDied","Data":"9e22e5e31f60c8023b63236694bf52afe6d97eabe71fa404eb0abe3d77bbbb01"}
Jul 17 21:57:09.656439 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:09.656107    2569 scope.go:117] "RemoveContainer" containerID="e890a7a2a46fe6655f8bdc4381938f617a933877976e42dfefdc24013dd68c9a"
Jul 17 21:57:09.656564 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:09.656547    2569 scope.go:117] "RemoveContainer" containerID="9e22e5e31f60c8023b63236694bf52afe6d97eabe71fa404eb0abe3d77bbbb01"
Jul 17 21:57:09.656811 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:57:09.656793    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz_odh-ai-gateway-infra(91f78a14-22f2-4e78-83e8-913dc57166b6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz" podUID="91f78a14-22f2-4e78-83e8-913dc57166b6"
Jul 17 21:57:10.680853 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:10.680820    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz"]
Jul 17 21:57:10.827924 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:10.827899    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz"
Jul 17 21:57:10.896716 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:10.896684    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91f78a14-22f2-4e78-83e8-913dc57166b6-maas-api-service-ca\") pod \"91f78a14-22f2-4e78-83e8-913dc57166b6\" (UID: \"91f78a14-22f2-4e78-83e8-913dc57166b6\") "
Jul 17 21:57:10.897062 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:10.897039    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f78a14-22f2-4e78-83e8-913dc57166b6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "91f78a14-22f2-4e78-83e8-913dc57166b6" (UID: "91f78a14-22f2-4e78-83e8-913dc57166b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:57:10.997548 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:10.997461    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91f78a14-22f2-4e78-83e8-913dc57166b6-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:57:11.650507 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:11.650476    2569 scope.go:117] "RemoveContainer" containerID="3bd17edca2539cdd990f32ad489af3eb22856bb540dbd7a36e8263f0ce30f149"
Jul 17 21:57:11.672777 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:11.672751    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz"
Jul 17 21:57:11.672893 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:11.672781    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz" event={"ID":"91f78a14-22f2-4e78-83e8-913dc57166b6","Type":"ContainerDied","Data":"e3df4da99f7be7209f16b172f2599c5485185de3062f69085d7c85a16f00d39f"}
Jul 17 21:57:11.672893 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:11.672846    2569 scope.go:117] "RemoveContainer" containerID="9e22e5e31f60c8023b63236694bf52afe6d97eabe71fa404eb0abe3d77bbbb01"
Jul 17 21:57:11.702444 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:11.702415    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz"]
Jul 17 21:57:11.705955 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:11.705930    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jstmz"]
Jul 17 21:57:12.657006 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:12.656924    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91f78a14-22f2-4e78-83e8-913dc57166b6" path="/var/lib/kubelet/pods/91f78a14-22f2-4e78-83e8-913dc57166b6/volumes"
Jul 17 21:57:12.681978 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:12.681945    2569 generic.go:356] "Generic (PLEG): container finished" podID="37670757-bf7a-4566-82ac-7f2879a837c6" containerID="ab4d648ce2f8923410d2a50595e400a320613c95ad948417a0c9ceb60c3910cd" exitCode=6
Jul 17 21:57:12.682140 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:12.681985    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7" event={"ID":"37670757-bf7a-4566-82ac-7f2879a837c6","Type":"ContainerDied","Data":"ab4d648ce2f8923410d2a50595e400a320613c95ad948417a0c9ceb60c3910cd"}
Jul 17 21:57:12.682140 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:12.682031    2569 scope.go:117] "RemoveContainer" containerID="3bd17edca2539cdd990f32ad489af3eb22856bb540dbd7a36e8263f0ce30f149"
Jul 17 21:57:12.682380 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:12.682361    2569 scope.go:117] "RemoveContainer" containerID="ab4d648ce2f8923410d2a50595e400a320613c95ad948417a0c9ceb60c3910cd"
Jul 17 21:57:12.682658 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:57:12.682635    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7_odh-ai-gateway-infra(37670757-bf7a-4566-82ac-7f2879a837c6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7" podUID="37670757-bf7a-4566-82ac-7f2879a837c6"
Jul 17 21:57:13.706540 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:13.706501    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7"]
Jul 17 21:57:13.869477 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:13.869450    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7"
Jul 17 21:57:13.924634 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:13.924601    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37670757-bf7a-4566-82ac-7f2879a837c6-maas-api-service-ca\") pod \"37670757-bf7a-4566-82ac-7f2879a837c6\" (UID: \"37670757-bf7a-4566-82ac-7f2879a837c6\") "
Jul 17 21:57:13.924997 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:13.924972    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37670757-bf7a-4566-82ac-7f2879a837c6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "37670757-bf7a-4566-82ac-7f2879a837c6" (UID: "37670757-bf7a-4566-82ac-7f2879a837c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:57:14.025749 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:14.025710    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37670757-bf7a-4566-82ac-7f2879a837c6-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:57:14.697057 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:14.697025    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7"
Jul 17 21:57:14.697235 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:14.697063    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7" event={"ID":"37670757-bf7a-4566-82ac-7f2879a837c6","Type":"ContainerDied","Data":"b0b987de2817da06e023eec0e0bbbcf620f328bf9265b63e3aac4aa5304f2a95"}
Jul 17 21:57:14.697235 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:14.697128    2569 scope.go:117] "RemoveContainer" containerID="ab4d648ce2f8923410d2a50595e400a320613c95ad948417a0c9ceb60c3910cd"
Jul 17 21:57:14.712706 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:14.712680    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7"]
Jul 17 21:57:14.719215 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:14.719189    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-fcdr7"]
Jul 17 21:57:15.312549 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:15.312513    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:57:16.657911 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:16.657879    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37670757-bf7a-4566-82ac-7f2879a837c6" path="/var/lib/kubelet/pods/37670757-bf7a-4566-82ac-7f2879a837c6/volumes"
Jul 17 21:57:18.660533 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:18.660493    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:20.814125 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.814087    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc"]
Jul 17 21:57:20.814666 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.814646    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f78a14-22f2-4e78-83e8-913dc57166b6" containerName="revoke-keys"
Jul 17 21:57:20.814753 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.814668    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f78a14-22f2-4e78-83e8-913dc57166b6" containerName="revoke-keys"
Jul 17 21:57:20.814753 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.814682    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37670757-bf7a-4566-82ac-7f2879a837c6" containerName="revoke-keys"
Jul 17 21:57:20.814753 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.814691    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="37670757-bf7a-4566-82ac-7f2879a837c6" containerName="revoke-keys"
Jul 17 21:57:20.814753 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.814706    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37670757-bf7a-4566-82ac-7f2879a837c6" containerName="revoke-keys"
Jul 17 21:57:20.814753 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.814714    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="37670757-bf7a-4566-82ac-7f2879a837c6" containerName="revoke-keys"
Jul 17 21:57:20.814753 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.814727    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f78a14-22f2-4e78-83e8-913dc57166b6" containerName="revoke-keys"
Jul 17 21:57:20.814753 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.814735    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f78a14-22f2-4e78-83e8-913dc57166b6" containerName="revoke-keys"
Jul 17 21:57:20.815100 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.814756    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f78a14-22f2-4e78-83e8-913dc57166b6" containerName="revoke-keys"
Jul 17 21:57:20.815100 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.814764    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f78a14-22f2-4e78-83e8-913dc57166b6" containerName="revoke-keys"
Jul 17 21:57:20.815100 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.814907    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f78a14-22f2-4e78-83e8-913dc57166b6" containerName="revoke-keys"
Jul 17 21:57:20.815100 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.814922    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="37670757-bf7a-4566-82ac-7f2879a837c6" containerName="revoke-keys"
Jul 17 21:57:20.815100 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.814933    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="37670757-bf7a-4566-82ac-7f2879a837c6" containerName="revoke-keys"
Jul 17 21:57:20.815100 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.814941    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="37670757-bf7a-4566-82ac-7f2879a837c6" containerName="revoke-keys"
Jul 17 21:57:20.815100 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.814950    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f78a14-22f2-4e78-83e8-913dc57166b6" containerName="revoke-keys"
Jul 17 21:57:20.819736 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.819713    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc"
Jul 17 21:57:20.822058 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.822035    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:57:20.822334 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.822036    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:57:20.825140 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.825115    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc"]
Jul 17 21:57:20.881741 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.881710    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd2248d-abc7-4a99-b32e-4190ac90484f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-jljmc\" (UID: \"3cd2248d-abc7-4a99-b32e-4190ac90484f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc"
Jul 17 21:57:20.982175 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.982138    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd2248d-abc7-4a99-b32e-4190ac90484f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-jljmc\" (UID: \"3cd2248d-abc7-4a99-b32e-4190ac90484f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc"
Jul 17 21:57:20.982852 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:20.982825    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd2248d-abc7-4a99-b32e-4190ac90484f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-jljmc\" (UID: \"3cd2248d-abc7-4a99-b32e-4190ac90484f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc"
Jul 17 21:57:21.131780 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:21.131736    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc"
Jul 17 21:57:21.473146 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:21.473114    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc"]
Jul 17 21:57:21.475598 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:57:21.475565    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd2248d_abc7_4a99_b32e_4190ac90484f.slice/crio-b750855093f63b135737fdaedf8af10bc8000cf6ae6cc38984b9e1a8ae9cebe5 WatchSource:0}: Error finding container b750855093f63b135737fdaedf8af10bc8000cf6ae6cc38984b9e1a8ae9cebe5: Status 404 returned error can't find the container with id b750855093f63b135737fdaedf8af10bc8000cf6ae6cc38984b9e1a8ae9cebe5
Jul 17 21:57:21.740599 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:21.740510    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc" event={"ID":"3cd2248d-abc7-4a99-b32e-4190ac90484f","Type":"ContainerStarted","Data":"b750855093f63b135737fdaedf8af10bc8000cf6ae6cc38984b9e1a8ae9cebe5"}
Jul 17 21:57:22.194956 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.194916    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t" podUID="0d3653f0-977e-42ff-9632-5edab244ce05" containerName="main" containerID="cri-o://5d4101c9ccaf84689204b2898e5b829921092d64de5a3769dfe82b31e814a7e3" gracePeriod=45
Jul 17 21:57:22.448200 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.448172    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:57:22.495586 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.495555    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3653f0-977e-42ff-9632-5edab244ce05-tls-certs\") pod \"0d3653f0-977e-42ff-9632-5edab244ce05\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") "
Jul 17 21:57:22.495786 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.495606    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-model-cache\") pod \"0d3653f0-977e-42ff-9632-5edab244ce05\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") "
Jul 17 21:57:22.495786 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.495628    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-home\") pod \"0d3653f0-977e-42ff-9632-5edab244ce05\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") "
Jul 17 21:57:22.495786 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.495742    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-dshm\") pod \"0d3653f0-977e-42ff-9632-5edab244ce05\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") "
Jul 17 21:57:22.495786 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.495781    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlfxn\" (UniqueName: \"kubernetes.io/projected/0d3653f0-977e-42ff-9632-5edab244ce05-kube-api-access-mlfxn\") pod \"0d3653f0-977e-42ff-9632-5edab244ce05\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") "
Jul 17 21:57:22.496032 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.495808    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-tmp-dir\") pod \"0d3653f0-977e-42ff-9632-5edab244ce05\" (UID: \"0d3653f0-977e-42ff-9632-5edab244ce05\") "
Jul 17 21:57:22.496032 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.495910    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d3653f0-977e-42ff-9632-5edab244ce05" (UID: "0d3653f0-977e-42ff-9632-5edab244ce05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:57:22.496032 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.495966    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-home" (OuterVolumeSpecName: "home") pod "0d3653f0-977e-42ff-9632-5edab244ce05" (UID: "0d3653f0-977e-42ff-9632-5edab244ce05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:57:22.496197 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.496142    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-model-cache\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:57:22.496197 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.496146    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d3653f0-977e-42ff-9632-5edab244ce05" (UID: "0d3653f0-977e-42ff-9632-5edab244ce05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:57:22.496197 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.496162    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-home\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:57:22.498026 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.497996    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3653f0-977e-42ff-9632-5edab244ce05-kube-api-access-mlfxn" (OuterVolumeSpecName: "kube-api-access-mlfxn") pod "0d3653f0-977e-42ff-9632-5edab244ce05" (UID: "0d3653f0-977e-42ff-9632-5edab244ce05"). InnerVolumeSpecName "kube-api-access-mlfxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:57:22.498292 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.498261    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-dshm" (OuterVolumeSpecName: "dshm") pod "0d3653f0-977e-42ff-9632-5edab244ce05" (UID: "0d3653f0-977e-42ff-9632-5edab244ce05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:57:22.498292 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.498261    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d3653f0-977e-42ff-9632-5edab244ce05-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d3653f0-977e-42ff-9632-5edab244ce05" (UID: "0d3653f0-977e-42ff-9632-5edab244ce05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:57:22.597575 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.597542    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-dshm\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:57:22.597575 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.597570    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlfxn\" (UniqueName: \"kubernetes.io/projected/0d3653f0-977e-42ff-9632-5edab244ce05-kube-api-access-mlfxn\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:57:22.597575 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.597580    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3653f0-977e-42ff-9632-5edab244ce05-tmp-dir\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:57:22.597819 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.597590    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3653f0-977e-42ff-9632-5edab244ce05-tls-certs\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:57:22.747856 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.747772    2569 generic.go:356] "Generic (PLEG): container finished" podID="0d3653f0-977e-42ff-9632-5edab244ce05" containerID="5d4101c9ccaf84689204b2898e5b829921092d64de5a3769dfe82b31e814a7e3" exitCode=0
Jul 17 21:57:22.747856 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.747814    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"
Jul 17 21:57:22.748086 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.747857    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t" event={"ID":"0d3653f0-977e-42ff-9632-5edab244ce05","Type":"ContainerDied","Data":"5d4101c9ccaf84689204b2898e5b829921092d64de5a3769dfe82b31e814a7e3"}
Jul 17 21:57:22.748086 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.747898    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t" event={"ID":"0d3653f0-977e-42ff-9632-5edab244ce05","Type":"ContainerDied","Data":"8a7e4466114069274b737d7071ca5c0d30e5ba4d8859e78b80d71e6fb58739a9"}
Jul 17 21:57:22.748086 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.747920    2569 scope.go:117] "RemoveContainer" containerID="5d4101c9ccaf84689204b2898e5b829921092d64de5a3769dfe82b31e814a7e3"
Jul 17 21:57:22.749679 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.749654    2569 generic.go:356] "Generic (PLEG): container finished" podID="3cd2248d-abc7-4a99-b32e-4190ac90484f" containerID="168459c2466bcb5aaf70043876d9ad3acaecc91afeb3eeaf03edaa7081073101" exitCode=6
Jul 17 21:57:22.749804 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.749716    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc" event={"ID":"3cd2248d-abc7-4a99-b32e-4190ac90484f","Type":"ContainerDied","Data":"168459c2466bcb5aaf70043876d9ad3acaecc91afeb3eeaf03edaa7081073101"}
Jul 17 21:57:22.749945 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.749923    2569 scope.go:117] "RemoveContainer" containerID="168459c2466bcb5aaf70043876d9ad3acaecc91afeb3eeaf03edaa7081073101"
Jul 17 21:57:22.757229 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.757207    2569 scope.go:117] "RemoveContainer" containerID="5d4101c9ccaf84689204b2898e5b829921092d64de5a3769dfe82b31e814a7e3"
Jul 17 21:57:22.757517 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:57:22.757492    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d4101c9ccaf84689204b2898e5b829921092d64de5a3769dfe82b31e814a7e3\": container with ID starting with 5d4101c9ccaf84689204b2898e5b829921092d64de5a3769dfe82b31e814a7e3 not found: ID does not exist" containerID="5d4101c9ccaf84689204b2898e5b829921092d64de5a3769dfe82b31e814a7e3"
Jul 17 21:57:22.757592 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.757532    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4101c9ccaf84689204b2898e5b829921092d64de5a3769dfe82b31e814a7e3"} err="failed to get container status \"5d4101c9ccaf84689204b2898e5b829921092d64de5a3769dfe82b31e814a7e3\": rpc error: code = NotFound desc = could not find container \"5d4101c9ccaf84689204b2898e5b829921092d64de5a3769dfe82b31e814a7e3\": container with ID starting with 5d4101c9ccaf84689204b2898e5b829921092d64de5a3769dfe82b31e814a7e3 not found: ID does not exist"
Jul 17 21:57:22.781111 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.781076    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"]
Jul 17 21:57:22.786362 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:22.786329    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ffd161/test-model-ffd161-kserve-69987cf49c-tfw7t"]
Jul 17 21:57:23.078357 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.078321    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp" podUID="89df7841-881a-4b99-9268-f3aa220816d4" containerName="main" containerID="cri-o://118b2048f948a6017a7c66226b7c39b084666a5888316c35009bf213564ccdff" gracePeriod=45
Jul 17 21:57:23.291265 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.291217    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:23.343664 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.343633    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:57:23.405494 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.405464    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-model-cache\") pod \"89df7841-881a-4b99-9268-f3aa220816d4\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") "
Jul 17 21:57:23.405694 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.405526    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-dshm\") pod \"89df7841-881a-4b99-9268-f3aa220816d4\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") "
Jul 17 21:57:23.405694 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.405575    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-tmp-dir\") pod \"89df7841-881a-4b99-9268-f3aa220816d4\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") "
Jul 17 21:57:23.405694 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.405596    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89df7841-881a-4b99-9268-f3aa220816d4-tls-certs\") pod \"89df7841-881a-4b99-9268-f3aa220816d4\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") "
Jul 17 21:57:23.405694 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.405639    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvs8q\" (UniqueName: \"kubernetes.io/projected/89df7841-881a-4b99-9268-f3aa220816d4-kube-api-access-jvs8q\") pod \"89df7841-881a-4b99-9268-f3aa220816d4\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") "
Jul 17 21:57:23.405694 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.405666    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-home\") pod \"89df7841-881a-4b99-9268-f3aa220816d4\" (UID: \"89df7841-881a-4b99-9268-f3aa220816d4\") "
Jul 17 21:57:23.405975 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.405728    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "89df7841-881a-4b99-9268-f3aa220816d4" (UID: "89df7841-881a-4b99-9268-f3aa220816d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:57:23.405975 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.405845    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89df7841-881a-4b99-9268-f3aa220816d4" (UID: "89df7841-881a-4b99-9268-f3aa220816d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:57:23.405975 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.405924    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-home" (OuterVolumeSpecName: "home") pod "89df7841-881a-4b99-9268-f3aa220816d4" (UID: "89df7841-881a-4b99-9268-f3aa220816d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:57:23.406093 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.406004    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-home\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:57:23.406093 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.406015    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-model-cache\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:57:23.406093 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.406024    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-tmp-dir\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:57:23.407904 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.407875    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89df7841-881a-4b99-9268-f3aa220816d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89df7841-881a-4b99-9268-f3aa220816d4" (UID: "89df7841-881a-4b99-9268-f3aa220816d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:57:23.408009 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.407907    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-dshm" (OuterVolumeSpecName: "dshm") pod "89df7841-881a-4b99-9268-f3aa220816d4" (UID: "89df7841-881a-4b99-9268-f3aa220816d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:57:23.408009 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.407959    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89df7841-881a-4b99-9268-f3aa220816d4-kube-api-access-jvs8q" (OuterVolumeSpecName: "kube-api-access-jvs8q") pod "89df7841-881a-4b99-9268-f3aa220816d4" (UID: "89df7841-881a-4b99-9268-f3aa220816d4"). InnerVolumeSpecName "kube-api-access-jvs8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:57:23.506586 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.506553    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvs8q\" (UniqueName: \"kubernetes.io/projected/89df7841-881a-4b99-9268-f3aa220816d4-kube-api-access-jvs8q\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:57:23.506586 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.506583    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89df7841-881a-4b99-9268-f3aa220816d4-dshm\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:57:23.506586 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.506594    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89df7841-881a-4b99-9268-f3aa220816d4-tls-certs\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:57:23.756948 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.756914    2569 generic.go:356] "Generic (PLEG): container finished" podID="89df7841-881a-4b99-9268-f3aa220816d4" containerID="118b2048f948a6017a7c66226b7c39b084666a5888316c35009bf213564ccdff" exitCode=0
Jul 17 21:57:23.757122 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.756950    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"
Jul 17 21:57:23.757122 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.756994    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp" event={"ID":"89df7841-881a-4b99-9268-f3aa220816d4","Type":"ContainerDied","Data":"118b2048f948a6017a7c66226b7c39b084666a5888316c35009bf213564ccdff"}
Jul 17 21:57:23.757122 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.757041    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp" event={"ID":"89df7841-881a-4b99-9268-f3aa220816d4","Type":"ContainerDied","Data":"07867618aee9798edb1722c597aec4c40ee3600cde706d189a87563da5520ac0"}
Jul 17 21:57:23.757122 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.757059    2569 scope.go:117] "RemoveContainer" containerID="118b2048f948a6017a7c66226b7c39b084666a5888316c35009bf213564ccdff"
Jul 17 21:57:23.760809 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.760484    2569 generic.go:356] "Generic (PLEG): container finished" podID="3cd2248d-abc7-4a99-b32e-4190ac90484f" containerID="f63ba41e361524e3c5969f732015bd1544c9761531b3e2a2e0142f9299e423d7" exitCode=6
Jul 17 21:57:23.760809 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.760643    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc" event={"ID":"3cd2248d-abc7-4a99-b32e-4190ac90484f","Type":"ContainerDied","Data":"f63ba41e361524e3c5969f732015bd1544c9761531b3e2a2e0142f9299e423d7"}
Jul 17 21:57:23.760994 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.760823    2569 scope.go:117] "RemoveContainer" containerID="f63ba41e361524e3c5969f732015bd1544c9761531b3e2a2e0142f9299e423d7"
Jul 17 21:57:23.761104 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:57:23.761078    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-jljmc_odh-ai-gateway-infra(3cd2248d-abc7-4a99-b32e-4190ac90484f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc" podUID="3cd2248d-abc7-4a99-b32e-4190ac90484f"
Jul 17 21:57:23.767971 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.767948    2569 scope.go:117] "RemoveContainer" containerID="118b2048f948a6017a7c66226b7c39b084666a5888316c35009bf213564ccdff"
Jul 17 21:57:23.768289 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:57:23.768268    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"118b2048f948a6017a7c66226b7c39b084666a5888316c35009bf213564ccdff\": container with ID starting with 118b2048f948a6017a7c66226b7c39b084666a5888316c35009bf213564ccdff not found: ID does not exist" containerID="118b2048f948a6017a7c66226b7c39b084666a5888316c35009bf213564ccdff"
Jul 17 21:57:23.768351 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.768298    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"118b2048f948a6017a7c66226b7c39b084666a5888316c35009bf213564ccdff"} err="failed to get container status \"118b2048f948a6017a7c66226b7c39b084666a5888316c35009bf213564ccdff\": rpc error: code = NotFound desc = could not find container \"118b2048f948a6017a7c66226b7c39b084666a5888316c35009bf213564ccdff\": container with ID starting with 118b2048f948a6017a7c66226b7c39b084666a5888316c35009bf213564ccdff not found: ID does not exist"
Jul 17 21:57:23.768351 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.768316    2569 scope.go:117] "RemoveContainer" containerID="168459c2466bcb5aaf70043876d9ad3acaecc91afeb3eeaf03edaa7081073101"
Jul 17 21:57:23.791922 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.791890    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"]
Jul 17 21:57:23.795218 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:23.795193    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-786043/test-model-786043-kserve-5fd8777dd9-cx7rp"]
Jul 17 21:57:24.657277 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:24.657237    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d3653f0-977e-42ff-9632-5edab244ce05" path="/var/lib/kubelet/pods/0d3653f0-977e-42ff-9632-5edab244ce05/volumes"
Jul 17 21:57:24.657702 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:24.657654    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89df7841-881a-4b99-9268-f3aa220816d4" path="/var/lib/kubelet/pods/89df7841-881a-4b99-9268-f3aa220816d4/volumes"
Jul 17 21:57:24.769061 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:24.769016    2569 scope.go:117] "RemoveContainer" containerID="f63ba41e361524e3c5969f732015bd1544c9761531b3e2a2e0142f9299e423d7"
Jul 17 21:57:24.769318 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:57:24.769296    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-jljmc_odh-ai-gateway-infra(3cd2248d-abc7-4a99-b32e-4190ac90484f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc" podUID="3cd2248d-abc7-4a99-b32e-4190ac90484f"
Jul 17 21:57:27.752652 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:27.752615    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t"]
Jul 17 21:57:27.753034 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:27.752997    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d3653f0-977e-42ff-9632-5edab244ce05" containerName="main"
Jul 17 21:57:27.753034 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:27.753008    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d3653f0-977e-42ff-9632-5edab244ce05" containerName="main"
Jul 17 21:57:27.753034 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:27.753018    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37670757-bf7a-4566-82ac-7f2879a837c6" containerName="revoke-keys"
Jul 17 21:57:27.753034 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:27.753024    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="37670757-bf7a-4566-82ac-7f2879a837c6" containerName="revoke-keys"
Jul 17 21:57:27.753169 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:27.753040    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89df7841-881a-4b99-9268-f3aa220816d4" containerName="main"
Jul 17 21:57:27.753169 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:27.753046    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="89df7841-881a-4b99-9268-f3aa220816d4" containerName="main"
Jul 17 21:57:27.753169 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:27.753109    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f78a14-22f2-4e78-83e8-913dc57166b6" containerName="revoke-keys"
Jul 17 21:57:27.753169 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:27.753117    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d3653f0-977e-42ff-9632-5edab244ce05" containerName="main"
Jul 17 21:57:27.753169 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:27.753126    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="89df7841-881a-4b99-9268-f3aa220816d4" containerName="main"
Jul 17 21:57:27.757552 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:27.757533    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t"
Jul 17 21:57:27.763373 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:27.763351    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t"]
Jul 17 21:57:27.845466 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:27.845431    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e8558d-5cd7-4586-8efc-66cf6c8e1964-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t\" (UID: \"f7e8558d-5cd7-4586-8efc-66cf6c8e1964\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t"
Jul 17 21:57:27.946660 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:27.946619    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e8558d-5cd7-4586-8efc-66cf6c8e1964-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t\" (UID: \"f7e8558d-5cd7-4586-8efc-66cf6c8e1964\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t"
Jul 17 21:57:27.947299 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:27.947279    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e8558d-5cd7-4586-8efc-66cf6c8e1964-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t\" (UID: \"f7e8558d-5cd7-4586-8efc-66cf6c8e1964\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t"
Jul 17 21:57:28.071462 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:28.071358    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t"
Jul 17 21:57:28.210341 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:28.208357    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t"]
Jul 17 21:57:28.320963 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:28.320920    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:57:28.794823 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:28.794737    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t" event={"ID":"f7e8558d-5cd7-4586-8efc-66cf6c8e1964","Type":"ContainerStarted","Data":"3ddaf09775469a9f42116d9ad1f852c4286f1c6830812c74ce3d969c581b76c4"}
Jul 17 21:57:28.794823 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:28.794788    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t" event={"ID":"f7e8558d-5cd7-4586-8efc-66cf6c8e1964","Type":"ContainerStarted","Data":"b7ef58c4ac32b586a4eea821c9d4937a0d3fbbd6a8531aa75dcbe15e7ce6093d"}
Jul 17 21:57:28.812126 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:28.812080    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t" podStartSLOduration=1.812066401 podStartE2EDuration="1.812066401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:57:28.809528842 +0000 UTC m=+2962.569095551" watchObservedRunningTime="2026-07-17 21:57:28.812066401 +0000 UTC m=+2962.571633095"
Jul 17 21:57:29.450921 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:29.450817    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:29.514090 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:29.514028    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:29.802066 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:29.801980    2569 generic.go:356] "Generic (PLEG): container finished" podID="f7e8558d-5cd7-4586-8efc-66cf6c8e1964" containerID="3ddaf09775469a9f42116d9ad1f852c4286f1c6830812c74ce3d969c581b76c4" exitCode=6
Jul 17 21:57:29.802446 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:29.802066    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t" event={"ID":"f7e8558d-5cd7-4586-8efc-66cf6c8e1964","Type":"ContainerDied","Data":"3ddaf09775469a9f42116d9ad1f852c4286f1c6830812c74ce3d969c581b76c4"}
Jul 17 21:57:29.802446 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:29.802244    2569 scope.go:117] "RemoveContainer" containerID="3ddaf09775469a9f42116d9ad1f852c4286f1c6830812c74ce3d969c581b76c4"
Jul 17 21:57:30.809646 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:30.809562    2569 generic.go:356] "Generic (PLEG): container finished" podID="f7e8558d-5cd7-4586-8efc-66cf6c8e1964" containerID="5962db4db25c77b0915f772fe8e52789006ac5543fdb029d850ef0b006443bc3" exitCode=6
Jul 17 21:57:30.810055 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:30.809639    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t" event={"ID":"f7e8558d-5cd7-4586-8efc-66cf6c8e1964","Type":"ContainerDied","Data":"5962db4db25c77b0915f772fe8e52789006ac5543fdb029d850ef0b006443bc3"}
Jul 17 21:57:30.810055 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:30.809692    2569 scope.go:117] "RemoveContainer" containerID="3ddaf09775469a9f42116d9ad1f852c4286f1c6830812c74ce3d969c581b76c4"
Jul 17 21:57:30.810055 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:30.809903    2569 scope.go:117] "RemoveContainer" containerID="5962db4db25c77b0915f772fe8e52789006ac5543fdb029d850ef0b006443bc3"
Jul 17 21:57:30.810192 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:57:30.810094    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t_odh-ai-gateway-infra(f7e8558d-5cd7-4586-8efc-66cf6c8e1964)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t" podUID="f7e8558d-5cd7-4586-8efc-66cf6c8e1964"
Jul 17 21:57:31.817112 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:31.817079    2569 scope.go:117] "RemoveContainer" containerID="5962db4db25c77b0915f772fe8e52789006ac5543fdb029d850ef0b006443bc3"
Jul 17 21:57:31.817593 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:57:31.817355    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t_odh-ai-gateway-infra(f7e8558d-5cd7-4586-8efc-66cf6c8e1964)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t" podUID="f7e8558d-5cd7-4586-8efc-66cf6c8e1964"
Jul 17 21:57:33.995643 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:33.995611    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:34.220219 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:34.220187    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:34.332923 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:34.332887    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-786043-gcxdq"]
Jul 17 21:57:34.337853 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:34.337834    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-786043-gcxdq"
Jul 17 21:57:34.342898 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:34.342867    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-786043-gcxdq"]
Jul 17 21:57:34.404464 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:34.404427    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ef5926-5fd2-4128-8c58-76dba88ccc77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-786043-gcxdq\" (UID: \"38ef5926-5fd2-4128-8c58-76dba88ccc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-786043-gcxdq"
Jul 17 21:57:34.505913 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:34.505876    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ef5926-5fd2-4128-8c58-76dba88ccc77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-786043-gcxdq\" (UID: \"38ef5926-5fd2-4128-8c58-76dba88ccc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-786043-gcxdq"
Jul 17 21:57:34.506536 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:34.506514    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ef5926-5fd2-4128-8c58-76dba88ccc77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-786043-gcxdq\" (UID: \"38ef5926-5fd2-4128-8c58-76dba88ccc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-786043-gcxdq"
Jul 17 21:57:34.650053 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:34.649957    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-786043-gcxdq"
Jul 17 21:57:34.801991 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:34.801959    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-786043-gcxdq"]
Jul 17 21:57:34.803687 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:57:34.803652    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ef5926_5fd2_4128_8c58_76dba88ccc77.slice/crio-ad6a3331f34642a4b8903e6161041e6ec5286bba5714348dcaaff046338f16c8 WatchSource:0}: Error finding container ad6a3331f34642a4b8903e6161041e6ec5286bba5714348dcaaff046338f16c8: Status 404 returned error can't find the container with id ad6a3331f34642a4b8903e6161041e6ec5286bba5714348dcaaff046338f16c8
Jul 17 21:57:34.836184 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:34.836155    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-786043-gcxdq" event={"ID":"38ef5926-5fd2-4128-8c58-76dba88ccc77","Type":"ContainerStarted","Data":"ad6a3331f34642a4b8903e6161041e6ec5286bba5714348dcaaff046338f16c8"}
Jul 17 21:57:34.982358 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:34.982305    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:35.080555 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:35.080528    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:35.843006 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:35.842970    2569 generic.go:356] "Generic (PLEG): container finished" podID="38ef5926-5fd2-4128-8c58-76dba88ccc77" containerID="4e7dc06f26893cdadd9668a92cebf5480e13f97ab6cf3ad5a894c63435ff4758" exitCode=0
Jul 17 21:57:35.843182 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:35.843056    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-786043-gcxdq" event={"ID":"38ef5926-5fd2-4128-8c58-76dba88ccc77","Type":"ContainerDied","Data":"4e7dc06f26893cdadd9668a92cebf5480e13f97ab6cf3ad5a894c63435ff4758"}
Jul 17 21:57:36.993024 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:36.993000    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-786043-gcxdq"
Jul 17 21:57:37.129820 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:37.129720    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ef5926-5fd2-4128-8c58-76dba88ccc77-maas-api-service-ca\") pod \"38ef5926-5fd2-4128-8c58-76dba88ccc77\" (UID: \"38ef5926-5fd2-4128-8c58-76dba88ccc77\") "
Jul 17 21:57:37.130093 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:37.130068    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ef5926-5fd2-4128-8c58-76dba88ccc77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "38ef5926-5fd2-4128-8c58-76dba88ccc77" (UID: "38ef5926-5fd2-4128-8c58-76dba88ccc77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:57:37.230549 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:37.230520    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ef5926-5fd2-4128-8c58-76dba88ccc77-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:57:37.855685 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:37.855644    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-786043-gcxdq" event={"ID":"38ef5926-5fd2-4128-8c58-76dba88ccc77","Type":"ContainerDied","Data":"ad6a3331f34642a4b8903e6161041e6ec5286bba5714348dcaaff046338f16c8"}
Jul 17 21:57:37.855685 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:37.855678    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-786043-gcxdq"
Jul 17 21:57:37.855896 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:37.855706    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad6a3331f34642a4b8903e6161041e6ec5286bba5714348dcaaff046338f16c8"
Jul 17 21:57:39.651133 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:39.651101    2569 scope.go:117] "RemoveContainer" containerID="f63ba41e361524e3c5969f732015bd1544c9761531b3e2a2e0142f9299e423d7"
Jul 17 21:57:40.876293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:40.876261    2569 generic.go:356] "Generic (PLEG): container finished" podID="3cd2248d-abc7-4a99-b32e-4190ac90484f" containerID="36874c6f253c5064ffc3c4f8eb874ce6c72801ff21aef2a3d3675698499cfdd2" exitCode=6
Jul 17 21:57:40.876740 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:40.876340    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc" event={"ID":"3cd2248d-abc7-4a99-b32e-4190ac90484f","Type":"ContainerDied","Data":"36874c6f253c5064ffc3c4f8eb874ce6c72801ff21aef2a3d3675698499cfdd2"}
Jul 17 21:57:40.876740 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:40.876421    2569 scope.go:117] "RemoveContainer" containerID="f63ba41e361524e3c5969f732015bd1544c9761531b3e2a2e0142f9299e423d7"
Jul 17 21:57:40.876825 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:40.876755    2569 scope.go:117] "RemoveContainer" containerID="36874c6f253c5064ffc3c4f8eb874ce6c72801ff21aef2a3d3675698499cfdd2"
Jul 17 21:57:40.877007 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:57:40.876987    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-jljmc_odh-ai-gateway-infra(3cd2248d-abc7-4a99-b32e-4190ac90484f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc" podUID="3cd2248d-abc7-4a99-b32e-4190ac90484f"
Jul 17 21:57:41.902223 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:41.902188    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc"]
Jul 17 21:57:42.045465 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:42.045441    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc"
Jul 17 21:57:42.173114 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:42.173021    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd2248d-abc7-4a99-b32e-4190ac90484f-maas-api-service-ca\") pod \"3cd2248d-abc7-4a99-b32e-4190ac90484f\" (UID: \"3cd2248d-abc7-4a99-b32e-4190ac90484f\") "
Jul 17 21:57:42.173522 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:42.173499    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd2248d-abc7-4a99-b32e-4190ac90484f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3cd2248d-abc7-4a99-b32e-4190ac90484f" (UID: "3cd2248d-abc7-4a99-b32e-4190ac90484f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:57:42.273606 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:42.273572    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd2248d-abc7-4a99-b32e-4190ac90484f-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:57:42.891049 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:42.891017    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc"
Jul 17 21:57:42.891049 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:42.891044    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc" event={"ID":"3cd2248d-abc7-4a99-b32e-4190ac90484f","Type":"ContainerDied","Data":"b750855093f63b135737fdaedf8af10bc8000cf6ae6cc38984b9e1a8ae9cebe5"}
Jul 17 21:57:42.891292 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:42.891108    2569 scope.go:117] "RemoveContainer" containerID="36874c6f253c5064ffc3c4f8eb874ce6c72801ff21aef2a3d3675698499cfdd2"
Jul 17 21:57:42.908300 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:42.908278    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc"]
Jul 17 21:57:42.914997 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:42.914947    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jljmc"]
Jul 17 21:57:44.656616 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:44.656585    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cd2248d-abc7-4a99-b32e-4190ac90484f" path="/var/lib/kubelet/pods/3cd2248d-abc7-4a99-b32e-4190ac90484f/volumes"
Jul 17 21:57:44.825624 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:44.825583    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:45.650761 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:45.650728    2569 scope.go:117] "RemoveContainer" containerID="5962db4db25c77b0915f772fe8e52789006ac5543fdb029d850ef0b006443bc3"
Jul 17 21:57:46.918227 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:46.918195    2569 generic.go:356] "Generic (PLEG): container finished" podID="f7e8558d-5cd7-4586-8efc-66cf6c8e1964" containerID="6f5ba3a06a699788039ecc1a274a425df9dff37359badaee13ee0d72667851af" exitCode=6
Jul 17 21:57:46.918624 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:46.918267    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t" event={"ID":"f7e8558d-5cd7-4586-8efc-66cf6c8e1964","Type":"ContainerDied","Data":"6f5ba3a06a699788039ecc1a274a425df9dff37359badaee13ee0d72667851af"}
Jul 17 21:57:46.918624 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:46.918340    2569 scope.go:117] "RemoveContainer" containerID="5962db4db25c77b0915f772fe8e52789006ac5543fdb029d850ef0b006443bc3"
Jul 17 21:57:46.918740 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:46.918707    2569 scope.go:117] "RemoveContainer" containerID="6f5ba3a06a699788039ecc1a274a425df9dff37359badaee13ee0d72667851af"
Jul 17 21:57:46.918931 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:57:46.918909    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t_odh-ai-gateway-infra(f7e8558d-5cd7-4586-8efc-66cf6c8e1964)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t" podUID="f7e8558d-5cd7-4586-8efc-66cf6c8e1964"
Jul 17 21:57:47.943697 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:47.943656    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t"]
Jul 17 21:57:48.086827 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:48.086800    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t"
Jul 17 21:57:48.213857 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:48.213781    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:57:48.228179 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:48.228154    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e8558d-5cd7-4586-8efc-66cf6c8e1964-maas-api-service-ca\") pod \"f7e8558d-5cd7-4586-8efc-66cf6c8e1964\" (UID: \"f7e8558d-5cd7-4586-8efc-66cf6c8e1964\") "
Jul 17 21:57:48.228525 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:48.228503    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e8558d-5cd7-4586-8efc-66cf6c8e1964-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7e8558d-5cd7-4586-8efc-66cf6c8e1964" (UID: "f7e8558d-5cd7-4586-8efc-66cf6c8e1964"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:57:48.329537 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:48.329506    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e8558d-5cd7-4586-8efc-66cf6c8e1964-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:57:48.936260 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:48.936182    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t"
Jul 17 21:57:48.936260 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:48.936195    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t" event={"ID":"f7e8558d-5cd7-4586-8efc-66cf6c8e1964","Type":"ContainerDied","Data":"b7ef58c4ac32b586a4eea821c9d4937a0d3fbbd6a8531aa75dcbe15e7ce6093d"}
Jul 17 21:57:48.936260 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:48.936251    2569 scope.go:117] "RemoveContainer" containerID="6f5ba3a06a699788039ecc1a274a425df9dff37359badaee13ee0d72667851af"
Jul 17 21:57:48.956229 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:48.956193    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t"]
Jul 17 21:57:48.976937 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:48.976901    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m2w6t"]
Jul 17 21:57:49.429199 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.429167    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:49.665096 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.665058    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc"]
Jul 17 21:57:49.665524 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.665511    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cd2248d-abc7-4a99-b32e-4190ac90484f" containerName="revoke-keys"
Jul 17 21:57:49.665573 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.665527    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cd2248d-abc7-4a99-b32e-4190ac90484f" containerName="revoke-keys"
Jul 17 21:57:49.665573 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.665536    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cd2248d-abc7-4a99-b32e-4190ac90484f" containerName="revoke-keys"
Jul 17 21:57:49.665573 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.665545    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cd2248d-abc7-4a99-b32e-4190ac90484f" containerName="revoke-keys"
Jul 17 21:57:49.665573 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.665558    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ef5926-5fd2-4128-8c58-76dba88ccc77" containerName="revoke-keys"
Jul 17 21:57:49.665573 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.665564    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ef5926-5fd2-4128-8c58-76dba88ccc77" containerName="revoke-keys"
Jul 17 21:57:49.665573 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.665573    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e8558d-5cd7-4586-8efc-66cf6c8e1964" containerName="revoke-keys"
Jul 17 21:57:49.665785 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.665579    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e8558d-5cd7-4586-8efc-66cf6c8e1964" containerName="revoke-keys"
Jul 17 21:57:49.665785 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.665596    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e8558d-5cd7-4586-8efc-66cf6c8e1964" containerName="revoke-keys"
Jul 17 21:57:49.665785 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.665602    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e8558d-5cd7-4586-8efc-66cf6c8e1964" containerName="revoke-keys"
Jul 17 21:57:49.665785 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.665670    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e8558d-5cd7-4586-8efc-66cf6c8e1964" containerName="revoke-keys"
Jul 17 21:57:49.665785 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.665677    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="38ef5926-5fd2-4128-8c58-76dba88ccc77" containerName="revoke-keys"
Jul 17 21:57:49.665785 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.665687    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e8558d-5cd7-4586-8efc-66cf6c8e1964" containerName="revoke-keys"
Jul 17 21:57:49.665785 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.665692    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cd2248d-abc7-4a99-b32e-4190ac90484f" containerName="revoke-keys"
Jul 17 21:57:49.665785 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.665702    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e8558d-5cd7-4586-8efc-66cf6c8e1964" containerName="revoke-keys"
Jul 17 21:57:49.665785 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.665711    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cd2248d-abc7-4a99-b32e-4190ac90484f" containerName="revoke-keys"
Jul 17 21:57:49.670296 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.670280    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc"
Jul 17 21:57:49.672743 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.672724    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:57:49.672883 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.672781    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:57:49.680011 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.679944    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc"]
Jul 17 21:57:49.742538 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.742492    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/331d9b19-1af3-4aac-8839-ac627289c16e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc\" (UID: \"331d9b19-1af3-4aac-8839-ac627289c16e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc"
Jul 17 21:57:49.843584 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.843549    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/331d9b19-1af3-4aac-8839-ac627289c16e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc\" (UID: \"331d9b19-1af3-4aac-8839-ac627289c16e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc"
Jul 17 21:57:49.844187 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.844161    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/331d9b19-1af3-4aac-8839-ac627289c16e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc\" (UID: \"331d9b19-1af3-4aac-8839-ac627289c16e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc"
Jul 17 21:57:49.981575 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:49.981494    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc"
Jul 17 21:57:50.126982 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:50.126953    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc"]
Jul 17 21:57:50.127844 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:57:50.127817    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod331d9b19_1af3_4aac_8839_ac627289c16e.slice/crio-928372dfedae7a7e13d3f9094a5af04f74173d0cfcbb34341e8e475e4f8de660 WatchSource:0}: Error finding container 928372dfedae7a7e13d3f9094a5af04f74173d0cfcbb34341e8e475e4f8de660: Status 404 returned error can't find the container with id 928372dfedae7a7e13d3f9094a5af04f74173d0cfcbb34341e8e475e4f8de660
Jul 17 21:57:50.656251 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:50.656169    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7e8558d-5cd7-4586-8efc-66cf6c8e1964" path="/var/lib/kubelet/pods/f7e8558d-5cd7-4586-8efc-66cf6c8e1964/volumes"
Jul 17 21:57:50.950414 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:50.950311    2569 generic.go:356] "Generic (PLEG): container finished" podID="331d9b19-1af3-4aac-8839-ac627289c16e" containerID="133afb24d2d1ff5efcab44c5f57c8a0084215f397268f0b54ec21989f4adc82e" exitCode=6
Jul 17 21:57:50.950578 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:50.950396    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc" event={"ID":"331d9b19-1af3-4aac-8839-ac627289c16e","Type":"ContainerDied","Data":"133afb24d2d1ff5efcab44c5f57c8a0084215f397268f0b54ec21989f4adc82e"}
Jul 17 21:57:50.950578 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:50.950461    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc" event={"ID":"331d9b19-1af3-4aac-8839-ac627289c16e","Type":"ContainerStarted","Data":"928372dfedae7a7e13d3f9094a5af04f74173d0cfcbb34341e8e475e4f8de660"}
Jul 17 21:57:50.950655 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:50.950575    2569 scope.go:117] "RemoveContainer" containerID="133afb24d2d1ff5efcab44c5f57c8a0084215f397268f0b54ec21989f4adc82e"
Jul 17 21:57:51.958000 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:51.957964    2569 generic.go:356] "Generic (PLEG): container finished" podID="331d9b19-1af3-4aac-8839-ac627289c16e" containerID="2c8c21b7dad0fc3674d5129cd1c074652200da6071e4cb90717e0c6ad0ac93e7" exitCode=6
Jul 17 21:57:51.958446 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:51.958010    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc" event={"ID":"331d9b19-1af3-4aac-8839-ac627289c16e","Type":"ContainerDied","Data":"2c8c21b7dad0fc3674d5129cd1c074652200da6071e4cb90717e0c6ad0ac93e7"}
Jul 17 21:57:51.958446 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:51.958070    2569 scope.go:117] "RemoveContainer" containerID="133afb24d2d1ff5efcab44c5f57c8a0084215f397268f0b54ec21989f4adc82e"
Jul 17 21:57:51.958446 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:51.958214    2569 scope.go:117] "RemoveContainer" containerID="2c8c21b7dad0fc3674d5129cd1c074652200da6071e4cb90717e0c6ad0ac93e7"
Jul 17 21:57:51.958618 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:57:51.958480    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc_odh-ai-gateway-infra(331d9b19-1af3-4aac-8839-ac627289c16e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc" podUID="331d9b19-1af3-4aac-8839-ac627289c16e"
Jul 17 21:57:52.967881 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:52.967851    2569 scope.go:117] "RemoveContainer" containerID="2c8c21b7dad0fc3674d5129cd1c074652200da6071e4cb90717e0c6ad0ac93e7"
Jul 17 21:57:52.968270 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:57:52.968068    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc_odh-ai-gateway-infra(331d9b19-1af3-4aac-8839-ac627289c16e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc" podUID="331d9b19-1af3-4aac-8839-ac627289c16e"
Jul 17 21:57:55.239147 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:55.239116    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:55.339273 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:57:55.339244    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:00.101845 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:00.101798    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:00.320143 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:00.320105    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:00.654221 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:00.654183    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:01.070395 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:01.070280    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:01.236134 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:01.236100    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:04.038039 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:04.037993    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:04.128396 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:04.128362    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:04.243868 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:04.243836    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:04.393667 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:04.393636    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:04.726585 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:04.726499    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:04.786600 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:04.786567    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:06.907982 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:06.907957    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:58:06.934977 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:06.934953    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 21:58:07.108266 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:07.108217    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:07.650250 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:07.650222    2569 scope.go:117] "RemoveContainer" containerID="2c8c21b7dad0fc3674d5129cd1c074652200da6071e4cb90717e0c6ad0ac93e7"
Jul 17 21:58:08.061105 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:08.061063    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc" event={"ID":"331d9b19-1af3-4aac-8839-ac627289c16e","Type":"ContainerStarted","Data":"0d86b29e0df5d4c789fb5dc06547a4a21a59461d6f96ff268e46610ebd27c926"}
Jul 17 21:58:08.079899 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:08.079853    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc" podStartSLOduration=19.079838182 podStartE2EDuration="19.079838182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:58:08.076742392 +0000 UTC m=+3001.836309087" watchObservedRunningTime="2026-07-17 21:58:08.079838182 +0000 UTC m=+3001.839404876"
Jul 17 21:58:08.460282 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:08.460248    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:08.897780 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:08.897746    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:09.069943 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:09.069906    2569 generic.go:356] "Generic (PLEG): container finished" podID="331d9b19-1af3-4aac-8839-ac627289c16e" containerID="0d86b29e0df5d4c789fb5dc06547a4a21a59461d6f96ff268e46610ebd27c926" exitCode=6
Jul 17 21:58:09.070370 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:09.069983    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc" event={"ID":"331d9b19-1af3-4aac-8839-ac627289c16e","Type":"ContainerDied","Data":"0d86b29e0df5d4c789fb5dc06547a4a21a59461d6f96ff268e46610ebd27c926"}
Jul 17 21:58:09.070370 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:09.070056    2569 scope.go:117] "RemoveContainer" containerID="2c8c21b7dad0fc3674d5129cd1c074652200da6071e4cb90717e0c6ad0ac93e7"
Jul 17 21:58:09.070523 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:09.070384    2569 scope.go:117] "RemoveContainer" containerID="0d86b29e0df5d4c789fb5dc06547a4a21a59461d6f96ff268e46610ebd27c926"
Jul 17 21:58:09.070636 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:58:09.070615    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc_odh-ai-gateway-infra(331d9b19-1af3-4aac-8839-ac627289c16e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc" podUID="331d9b19-1af3-4aac-8839-ac627289c16e"
Jul 17 21:58:09.097854 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:09.097823    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc"]
Jul 17 21:58:09.197348 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:09.197272    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:09.521248 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:09.521166    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:58:09.606594 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:09.606564    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:10.217435 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:10.217386    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc"
Jul 17 21:58:10.327467 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:10.327431    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/331d9b19-1af3-4aac-8839-ac627289c16e-maas-api-service-ca\") pod \"331d9b19-1af3-4aac-8839-ac627289c16e\" (UID: \"331d9b19-1af3-4aac-8839-ac627289c16e\") "
Jul 17 21:58:10.327680 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:10.327660    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/331d9b19-1af3-4aac-8839-ac627289c16e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "331d9b19-1af3-4aac-8839-ac627289c16e" (UID: "331d9b19-1af3-4aac-8839-ac627289c16e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:58:10.428062 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:10.428027    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/331d9b19-1af3-4aac-8839-ac627289c16e-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:58:11.084931 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:11.084900    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc"
Jul 17 21:58:11.085088 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:11.084929    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc" event={"ID":"331d9b19-1af3-4aac-8839-ac627289c16e","Type":"ContainerDied","Data":"928372dfedae7a7e13d3f9094a5af04f74173d0cfcbb34341e8e475e4f8de660"}
Jul 17 21:58:11.085088 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:11.084990    2569 scope.go:117] "RemoveContainer" containerID="0d86b29e0df5d4c789fb5dc06547a4a21a59461d6f96ff268e46610ebd27c926"
Jul 17 21:58:11.099879 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:11.099852    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc"]
Jul 17 21:58:11.105494 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:11.105471    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-shrnc"]
Jul 17 21:58:12.573762 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.573720    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:12.660167 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.660125    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="331d9b19-1af3-4aac-8839-ac627289c16e" path="/var/lib/kubelet/pods/331d9b19-1af3-4aac-8839-ac627289c16e/volumes"
Jul 17 21:58:12.660602 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.660577    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"]
Jul 17 21:58:12.661312 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.661288    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="331d9b19-1af3-4aac-8839-ac627289c16e" containerName="revoke-keys"
Jul 17 21:58:12.661512 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.661496    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="331d9b19-1af3-4aac-8839-ac627289c16e" containerName="revoke-keys"
Jul 17 21:58:12.661635 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.661622    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cd2248d-abc7-4a99-b32e-4190ac90484f" containerName="revoke-keys"
Jul 17 21:58:12.661731 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.661720    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cd2248d-abc7-4a99-b32e-4190ac90484f" containerName="revoke-keys"
Jul 17 21:58:12.661837 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.661826    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e8558d-5cd7-4586-8efc-66cf6c8e1964" containerName="revoke-keys"
Jul 17 21:58:12.661918 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.661908    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e8558d-5cd7-4586-8efc-66cf6c8e1964" containerName="revoke-keys"
Jul 17 21:58:12.662037 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.662027    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="331d9b19-1af3-4aac-8839-ac627289c16e" containerName="revoke-keys"
Jul 17 21:58:12.662123 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.662113    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="331d9b19-1af3-4aac-8839-ac627289c16e" containerName="revoke-keys"
Jul 17 21:58:12.662314 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.662300    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="331d9b19-1af3-4aac-8839-ac627289c16e" containerName="revoke-keys"
Jul 17 21:58:12.662615 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.662392    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="331d9b19-1af3-4aac-8839-ac627289c16e" containerName="revoke-keys"
Jul 17 21:58:12.662812 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.662775    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="331d9b19-1af3-4aac-8839-ac627289c16e" containerName="revoke-keys"
Jul 17 21:58:12.662952 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.662938    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cd2248d-abc7-4a99-b32e-4190ac90484f" containerName="revoke-keys"
Jul 17 21:58:12.664442 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.662715    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:12.667839 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.667819    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"
Jul 17 21:58:12.669881 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.669862    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mbgfd\""
Jul 17 21:58:12.670064 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.669953    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-dd0e35-serving-cert\""
Jul 17 21:58:12.670351 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.670334    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:58:12.670445 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.670359    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 21:58:12.676811 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.676625    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"]
Jul 17 21:58:12.738170 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.738133    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:12.742478 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.742451    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65897541-c878-4315-a499-4691b3aaa949-maas-api-tls\") pod \"maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls\" (UID: \"65897541-c878-4315-a499-4691b3aaa949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"
Jul 17 21:58:12.742598 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.742479    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5xls\" (UniqueName: \"kubernetes.io/projected/65897541-c878-4315-a499-4691b3aaa949-kube-api-access-t5xls\") pod \"maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls\" (UID: \"65897541-c878-4315-a499-4691b3aaa949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"
Jul 17 21:58:12.742598 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.742555    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65897541-c878-4315-a499-4691b3aaa949-openshift-service-ca\") pod \"maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls\" (UID: \"65897541-c878-4315-a499-4691b3aaa949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"
Jul 17 21:58:12.842990 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.842959    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65897541-c878-4315-a499-4691b3aaa949-maas-api-tls\") pod \"maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls\" (UID: \"65897541-c878-4315-a499-4691b3aaa949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"
Jul 17 21:58:12.843156 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.842995    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5xls\" (UniqueName: \"kubernetes.io/projected/65897541-c878-4315-a499-4691b3aaa949-kube-api-access-t5xls\") pod \"maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls\" (UID: \"65897541-c878-4315-a499-4691b3aaa949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"
Jul 17 21:58:12.843156 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.843064    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65897541-c878-4315-a499-4691b3aaa949-openshift-service-ca\") pod \"maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls\" (UID: \"65897541-c878-4315-a499-4691b3aaa949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"
Jul 17 21:58:12.843796 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.843774    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65897541-c878-4315-a499-4691b3aaa949-openshift-service-ca\") pod \"maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls\" (UID: \"65897541-c878-4315-a499-4691b3aaa949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"
Jul 17 21:58:12.845390 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.845366    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65897541-c878-4315-a499-4691b3aaa949-maas-api-tls\") pod \"maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls\" (UID: \"65897541-c878-4315-a499-4691b3aaa949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"
Jul 17 21:58:12.852497 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.852470    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5xls\" (UniqueName: \"kubernetes.io/projected/65897541-c878-4315-a499-4691b3aaa949-kube-api-access-t5xls\") pod \"maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls\" (UID: \"65897541-c878-4315-a499-4691b3aaa949\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"
Jul 17 21:58:12.872827 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.872785    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:12.981493 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:12.981461    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"
Jul 17 21:58:13.122446 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:13.122392    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"]
Jul 17 21:58:13.123856 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:58:13.123827    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65897541_c878_4315_a499_4691b3aaa949.slice/crio-0c14a3687ee88c4f0caf80ec2fcfbd590803bcbff537a2bf5c2dae192c4bbeb6 WatchSource:0}: Error finding container 0c14a3687ee88c4f0caf80ec2fcfbd590803bcbff537a2bf5c2dae192c4bbeb6: Status 404 returned error can't find the container with id 0c14a3687ee88c4f0caf80ec2fcfbd590803bcbff537a2bf5c2dae192c4bbeb6
Jul 17 21:58:13.125243 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:13.125222    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 21:58:13.244921 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:13.244886    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:13.345051 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:13.345017    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:14.110989 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:14.110879    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls" event={"ID":"65897541-c878-4315-a499-4691b3aaa949","Type":"ContainerStarted","Data":"0c14a3687ee88c4f0caf80ec2fcfbd590803bcbff537a2bf5c2dae192c4bbeb6"}
Jul 17 21:58:16.126831 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:16.126791    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls" event={"ID":"65897541-c878-4315-a499-4691b3aaa949","Type":"ContainerStarted","Data":"b1322b948fa657f36ceb183f5632dc12fc9408cdf1d86d68a59e01212aa175d6"}
Jul 17 21:58:16.127265 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:16.126903    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"
Jul 17 21:58:16.145271 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:16.145217    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls" podStartSLOduration=2.039830801 podStartE2EDuration="4.145200891s" totalImagesPullingTime="2.10537009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:58:16.144763345 +0000 UTC m=+3009.904330054" watchObservedRunningTime="2026-07-17 21:58:16.145200891 +0000 UTC m=+3009.904767584"
Jul 17 21:58:17.052507 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:17.052471    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:22.138033 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:22.137998    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"
Jul 17 21:58:23.717145 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:23.717109    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:58:25.533763 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:25.533730    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:25.635469 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:25.635440    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:34.188711 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:34.188669    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:34.299730 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:34.299691    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:35.580357 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:35.580317    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:35.680364 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:35.680331    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:35.853614 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:35.853536    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:39.819603 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:39.819561    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:58:40.078247 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:40.078165    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:45.936862 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:45.936830    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:46.023489 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:46.023452    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:50.783995 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:50.783961    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:50.994156 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:50.994120    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:51.856174 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:51.856142    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:51.957325 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:58:51.957295    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:01.579758 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:01.579723    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"]
Jul 17 21:59:01.580355 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:01.580057    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls" podUID="65897541-c878-4315-a499-4691b3aaa949" containerName="maas-api" containerID="cri-o://b1322b948fa657f36ceb183f5632dc12fc9408cdf1d86d68a59e01212aa175d6" gracePeriod=30
Jul 17 21:59:01.605368 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:01.605343    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:01.822809 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:01.822783    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"
Jul 17 21:59:01.989964 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:01.989931    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65897541-c878-4315-a499-4691b3aaa949-openshift-service-ca\") pod \"65897541-c878-4315-a499-4691b3aaa949\" (UID: \"65897541-c878-4315-a499-4691b3aaa949\") "
Jul 17 21:59:01.990144 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:01.990022    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65897541-c878-4315-a499-4691b3aaa949-maas-api-tls\") pod \"65897541-c878-4315-a499-4691b3aaa949\" (UID: \"65897541-c878-4315-a499-4691b3aaa949\") "
Jul 17 21:59:01.990144 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:01.990042    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5xls\" (UniqueName: \"kubernetes.io/projected/65897541-c878-4315-a499-4691b3aaa949-kube-api-access-t5xls\") pod \"65897541-c878-4315-a499-4691b3aaa949\" (UID: \"65897541-c878-4315-a499-4691b3aaa949\") "
Jul 17 21:59:01.990357 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:01.990331    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65897541-c878-4315-a499-4691b3aaa949-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "65897541-c878-4315-a499-4691b3aaa949" (UID: "65897541-c878-4315-a499-4691b3aaa949"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:59:01.992281 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:01.992249    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65897541-c878-4315-a499-4691b3aaa949-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "65897541-c878-4315-a499-4691b3aaa949" (UID: "65897541-c878-4315-a499-4691b3aaa949"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:59:01.992389 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:01.992302    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65897541-c878-4315-a499-4691b3aaa949-kube-api-access-t5xls" (OuterVolumeSpecName: "kube-api-access-t5xls") pod "65897541-c878-4315-a499-4691b3aaa949" (UID: "65897541-c878-4315-a499-4691b3aaa949"). InnerVolumeSpecName "kube-api-access-t5xls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:59:02.090880 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:02.090835    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65897541-c878-4315-a499-4691b3aaa949-openshift-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:59:02.090880 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:02.090873    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65897541-c878-4315-a499-4691b3aaa949-maas-api-tls\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:59:02.090880 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:02.090882    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5xls\" (UniqueName: \"kubernetes.io/projected/65897541-c878-4315-a499-4691b3aaa949-kube-api-access-t5xls\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:59:02.402173 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:02.402135    2569 generic.go:356] "Generic (PLEG): container finished" podID="65897541-c878-4315-a499-4691b3aaa949" containerID="b1322b948fa657f36ceb183f5632dc12fc9408cdf1d86d68a59e01212aa175d6" exitCode=0
Jul 17 21:59:02.404196 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:02.402179    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"
Jul 17 21:59:02.404196 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:02.402217    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls" event={"ID":"65897541-c878-4315-a499-4691b3aaa949","Type":"ContainerDied","Data":"b1322b948fa657f36ceb183f5632dc12fc9408cdf1d86d68a59e01212aa175d6"}
Jul 17 21:59:02.404196 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:02.402271    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls" event={"ID":"65897541-c878-4315-a499-4691b3aaa949","Type":"ContainerDied","Data":"0c14a3687ee88c4f0caf80ec2fcfbd590803bcbff537a2bf5c2dae192c4bbeb6"}
Jul 17 21:59:02.404196 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:02.402295    2569 scope.go:117] "RemoveContainer" containerID="b1322b948fa657f36ceb183f5632dc12fc9408cdf1d86d68a59e01212aa175d6"
Jul 17 21:59:02.411978 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:02.411961    2569 scope.go:117] "RemoveContainer" containerID="b1322b948fa657f36ceb183f5632dc12fc9408cdf1d86d68a59e01212aa175d6"
Jul 17 21:59:02.412234 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:59:02.412217    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1322b948fa657f36ceb183f5632dc12fc9408cdf1d86d68a59e01212aa175d6\": container with ID starting with b1322b948fa657f36ceb183f5632dc12fc9408cdf1d86d68a59e01212aa175d6 not found: ID does not exist" containerID="b1322b948fa657f36ceb183f5632dc12fc9408cdf1d86d68a59e01212aa175d6"
Jul 17 21:59:02.412278 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:02.412242    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1322b948fa657f36ceb183f5632dc12fc9408cdf1d86d68a59e01212aa175d6"} err="failed to get container status \"b1322b948fa657f36ceb183f5632dc12fc9408cdf1d86d68a59e01212aa175d6\": rpc error: code = NotFound desc = could not find container \"b1322b948fa657f36ceb183f5632dc12fc9408cdf1d86d68a59e01212aa175d6\": container with ID starting with b1322b948fa657f36ceb183f5632dc12fc9408cdf1d86d68a59e01212aa175d6 not found: ID does not exist"
Jul 17 21:59:02.423011 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:02.422988    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"]
Jul 17 21:59:02.426910 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:02.426887    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd0e35-5cf6c769fd-7mhls"]
Jul 17 21:59:02.656391 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:02.656308    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65897541-c878-4315-a499-4691b3aaa949" path="/var/lib/kubelet/pods/65897541-c878-4315-a499-4691b3aaa949/volumes"
Jul 17 21:59:03.061484 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:03.061376    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:03.165242 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:03.165204    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:04.920080 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:04.920041    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:59:06.207601 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:06.207568    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:12.144434 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:12.144377    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:12.244981 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:12.244949    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:16.885489 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:16.885451    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:17.108220 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:17.108183    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:17.442023 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:17.441991    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:17.764831 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:17.764730    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:18.373063 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:18.373018    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:20.368599 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:20.368551    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:20.460490 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:20.460446    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:20.579104 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:20.579073    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:20.661136 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:20.661049    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:20.914675 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:20.914591    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8"]
Jul 17 21:59:20.915059 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:20.915043    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65897541-c878-4315-a499-4691b3aaa949" containerName="maas-api"
Jul 17 21:59:20.915103 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:20.915062    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="65897541-c878-4315-a499-4691b3aaa949" containerName="maas-api"
Jul 17 21:59:20.915103 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:20.915080    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="331d9b19-1af3-4aac-8839-ac627289c16e" containerName="revoke-keys"
Jul 17 21:59:20.915103 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:20.915086    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="331d9b19-1af3-4aac-8839-ac627289c16e" containerName="revoke-keys"
Jul 17 21:59:20.915202 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:20.915168    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="65897541-c878-4315-a499-4691b3aaa949" containerName="maas-api"
Jul 17 21:59:20.918262 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:20.918245    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8"
Jul 17 21:59:20.920296 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:20.920269    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:59:20.920436 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:20.920325    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:59:20.927810 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:20.927785    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8"]
Jul 17 21:59:20.936800 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:20.936775    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4697739a-79e3-41fc-9b47-2e8993968f40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-vxbm8\" (UID: \"4697739a-79e3-41fc-9b47-2e8993968f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8"
Jul 17 21:59:21.037298 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:21.037264    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4697739a-79e3-41fc-9b47-2e8993968f40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-vxbm8\" (UID: \"4697739a-79e3-41fc-9b47-2e8993968f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8"
Jul 17 21:59:21.038003 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:21.037973    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4697739a-79e3-41fc-9b47-2e8993968f40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-vxbm8\" (UID: \"4697739a-79e3-41fc-9b47-2e8993968f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8"
Jul 17 21:59:21.068459 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:21.068417    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:21.168727 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:21.168647    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:21.229677 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:21.229646    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8"
Jul 17 21:59:21.366071 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:21.366038    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8"]
Jul 17 21:59:21.368739 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:59:21.368701    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4697739a_79e3_41fc_9b47_2e8993968f40.slice/crio-9d9a9efbbb3097c0533f3faef9e851570a109eea31dc4509ffd70ec79e90164b WatchSource:0}: Error finding container 9d9a9efbbb3097c0533f3faef9e851570a109eea31dc4509ffd70ec79e90164b: Status 404 returned error can't find the container with id 9d9a9efbbb3097c0533f3faef9e851570a109eea31dc4509ffd70ec79e90164b
Jul 17 21:59:21.530525 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:21.530250    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8" event={"ID":"4697739a-79e3-41fc-9b47-2e8993968f40","Type":"ContainerStarted","Data":"9d9a9efbbb3097c0533f3faef9e851570a109eea31dc4509ffd70ec79e90164b"}
Jul 17 21:59:22.538831 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:22.538792    2569 generic.go:356] "Generic (PLEG): container finished" podID="4697739a-79e3-41fc-9b47-2e8993968f40" containerID="e7dd58fec025c5fe4f7f2c3c70373309da3f2a9db61fee5296c3da4d0b184ca6" exitCode=6
Jul 17 21:59:22.539293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:22.538853    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8" event={"ID":"4697739a-79e3-41fc-9b47-2e8993968f40","Type":"ContainerDied","Data":"e7dd58fec025c5fe4f7f2c3c70373309da3f2a9db61fee5296c3da4d0b184ca6"}
Jul 17 21:59:22.539293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:22.539094    2569 scope.go:117] "RemoveContainer" containerID="e7dd58fec025c5fe4f7f2c3c70373309da3f2a9db61fee5296c3da4d0b184ca6"
Jul 17 21:59:23.545907 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:23.545819    2569 generic.go:356] "Generic (PLEG): container finished" podID="4697739a-79e3-41fc-9b47-2e8993968f40" containerID="8c86a820f6ca6fc8e06ac12b1625261f8f5c3cec948e063de31d9b35d2fbd5ec" exitCode=6
Jul 17 21:59:23.545907 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:23.545891    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8" event={"ID":"4697739a-79e3-41fc-9b47-2e8993968f40","Type":"ContainerDied","Data":"8c86a820f6ca6fc8e06ac12b1625261f8f5c3cec948e063de31d9b35d2fbd5ec"}
Jul 17 21:59:23.546331 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:23.545952    2569 scope.go:117] "RemoveContainer" containerID="e7dd58fec025c5fe4f7f2c3c70373309da3f2a9db61fee5296c3da4d0b184ca6"
Jul 17 21:59:23.546331 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:23.546158    2569 scope.go:117] "RemoveContainer" containerID="8c86a820f6ca6fc8e06ac12b1625261f8f5c3cec948e063de31d9b35d2fbd5ec"
Jul 17 21:59:23.546416 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:59:23.546376    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-vxbm8_odh-ai-gateway-infra(4697739a-79e3-41fc-9b47-2e8993968f40)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8" podUID="4697739a-79e3-41fc-9b47-2e8993968f40"
Jul 17 21:59:24.553258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:24.553232    2569 scope.go:117] "RemoveContainer" containerID="8c86a820f6ca6fc8e06ac12b1625261f8f5c3cec948e063de31d9b35d2fbd5ec"
Jul 17 21:59:24.553656 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:59:24.553445    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-vxbm8_odh-ai-gateway-infra(4697739a-79e3-41fc-9b47-2e8993968f40)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8" podUID="4697739a-79e3-41fc-9b47-2e8993968f40"
Jul 17 21:59:24.857728 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:24.857691    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:25.172842 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:25.172754    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:25.577991 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:25.577899    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:26.054078 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:26.054050    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:26.415748 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:26.415714    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:59:27.857724 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:27.857688    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj"]
Jul 17 21:59:27.862647 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:27.862624    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj"
Jul 17 21:59:27.868377 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:27.868350    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj"]
Jul 17 21:59:27.890708 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:27.890674    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b6d4391-ecd7-4b1b-937b-3763eae81905-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj\" (UID: \"3b6d4391-ecd7-4b1b-937b-3763eae81905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj"
Jul 17 21:59:27.991928 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:27.991890    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b6d4391-ecd7-4b1b-937b-3763eae81905-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj\" (UID: \"3b6d4391-ecd7-4b1b-937b-3763eae81905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj"
Jul 17 21:59:27.992575 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:27.992552    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b6d4391-ecd7-4b1b-937b-3763eae81905-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj\" (UID: \"3b6d4391-ecd7-4b1b-937b-3763eae81905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj"
Jul 17 21:59:28.174613 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:28.174514    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj"
Jul 17 21:59:28.315180 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:59:28.315148    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6d4391_ecd7_4b1b_937b_3763eae81905.slice/crio-52cfa7574b54c0dcf7eb09d53aa0b8475b82c91ce2360635e89de03ba6d6cfa7 WatchSource:0}: Error finding container 52cfa7574b54c0dcf7eb09d53aa0b8475b82c91ce2360635e89de03ba6d6cfa7: Status 404 returned error can't find the container with id 52cfa7574b54c0dcf7eb09d53aa0b8475b82c91ce2360635e89de03ba6d6cfa7
Jul 17 21:59:28.315180 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:28.315155    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj"]
Jul 17 21:59:28.581772 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:28.581740    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj" event={"ID":"3b6d4391-ecd7-4b1b-937b-3763eae81905","Type":"ContainerStarted","Data":"52cfa7574b54c0dcf7eb09d53aa0b8475b82c91ce2360635e89de03ba6d6cfa7"}
Jul 17 21:59:28.962188 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:28.962154    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:29.067795 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.067734    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:29.071046 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.071017    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"]
Jul 17 21:59:29.075269 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.075242    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"
Jul 17 21:59:29.077258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.077233    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-051ea5-serving-cert\""
Jul 17 21:59:29.077356 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.077288    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mbgfd\""
Jul 17 21:59:29.079342 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.079325    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 21:59:29.084436 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.084388    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"]
Jul 17 21:59:29.102945 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.102912    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67rpc\" (UniqueName: \"kubernetes.io/projected/4788f201-3d97-4c2e-ba51-7a38df63b411-kube-api-access-67rpc\") pod \"maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9\" (UID: \"4788f201-3d97-4c2e-ba51-7a38df63b411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"
Jul 17 21:59:29.103092 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.102974    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4788f201-3d97-4c2e-ba51-7a38df63b411-openshift-service-ca\") pod \"maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9\" (UID: \"4788f201-3d97-4c2e-ba51-7a38df63b411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"
Jul 17 21:59:29.103092 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.103035    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4788f201-3d97-4c2e-ba51-7a38df63b411-maas-api-tls\") pod \"maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9\" (UID: \"4788f201-3d97-4c2e-ba51-7a38df63b411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"
Jul 17 21:59:29.130923 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.130893    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:29.204243 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.204206    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4788f201-3d97-4c2e-ba51-7a38df63b411-openshift-service-ca\") pod \"maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9\" (UID: \"4788f201-3d97-4c2e-ba51-7a38df63b411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"
Jul 17 21:59:29.204427 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.204264    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4788f201-3d97-4c2e-ba51-7a38df63b411-maas-api-tls\") pod \"maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9\" (UID: \"4788f201-3d97-4c2e-ba51-7a38df63b411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"
Jul 17 21:59:29.204427 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.204361    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67rpc\" (UniqueName: \"kubernetes.io/projected/4788f201-3d97-4c2e-ba51-7a38df63b411-kube-api-access-67rpc\") pod \"maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9\" (UID: \"4788f201-3d97-4c2e-ba51-7a38df63b411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"
Jul 17 21:59:29.204503 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:59:29.204451    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-serving-cert: secret "maas-api-e2e-shared-b-051ea5-serving-cert" not found
Jul 17 21:59:29.204550 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:59:29.204538    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4788f201-3d97-4c2e-ba51-7a38df63b411-maas-api-tls podName:4788f201-3d97-4c2e-ba51-7a38df63b411 nodeName:}" failed. No retries permitted until 2026-07-17 21:59:29.704518618 +0000 UTC m=+3083.464085290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4788f201-3d97-4c2e-ba51-7a38df63b411-maas-api-tls") pod "maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9" (UID: "4788f201-3d97-4c2e-ba51-7a38df63b411") : secret "maas-api-e2e-shared-b-051ea5-serving-cert" not found
Jul 17 21:59:29.204978 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.204959    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4788f201-3d97-4c2e-ba51-7a38df63b411-openshift-service-ca\") pod \"maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9\" (UID: \"4788f201-3d97-4c2e-ba51-7a38df63b411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"
Jul 17 21:59:29.212598 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.212534    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67rpc\" (UniqueName: \"kubernetes.io/projected/4788f201-3d97-4c2e-ba51-7a38df63b411-kube-api-access-67rpc\") pod \"maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9\" (UID: \"4788f201-3d97-4c2e-ba51-7a38df63b411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"
Jul 17 21:59:29.367138 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.367107    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:29.593428 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.592722    2569 generic.go:356] "Generic (PLEG): container finished" podID="3b6d4391-ecd7-4b1b-937b-3763eae81905" containerID="f9914d275170c653555210d12184416e38be14d4b53dede289d14b2592908d17" exitCode=6
Jul 17 21:59:29.593428 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.592868    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj" event={"ID":"3b6d4391-ecd7-4b1b-937b-3763eae81905","Type":"ContainerDied","Data":"f9914d275170c653555210d12184416e38be14d4b53dede289d14b2592908d17"}
Jul 17 21:59:29.593428 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.593172    2569 scope.go:117] "RemoveContainer" containerID="f9914d275170c653555210d12184416e38be14d4b53dede289d14b2592908d17"
Jul 17 21:59:29.700689 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.700651    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:29.709248 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.709211    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4788f201-3d97-4c2e-ba51-7a38df63b411-maas-api-tls\") pod \"maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9\" (UID: \"4788f201-3d97-4c2e-ba51-7a38df63b411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"
Jul 17 21:59:29.712022 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.711996    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4788f201-3d97-4c2e-ba51-7a38df63b411-maas-api-tls\") pod \"maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9\" (UID: \"4788f201-3d97-4c2e-ba51-7a38df63b411\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"
Jul 17 21:59:29.803521 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.803481    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:29.994558 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:29.994523    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"
Jul 17 21:59:30.156263 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:30.156234    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"]
Jul 17 21:59:30.158691 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:59:30.158657    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4788f201_3d97_4c2e_ba51_7a38df63b411.slice/crio-eba49c838cdbee325434ae41b334b396687461ad8f8921b9ecd61b5cba7f836e WatchSource:0}: Error finding container eba49c838cdbee325434ae41b334b396687461ad8f8921b9ecd61b5cba7f836e: Status 404 returned error can't find the container with id eba49c838cdbee325434ae41b334b396687461ad8f8921b9ecd61b5cba7f836e
Jul 17 21:59:30.600423 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:30.600373    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9" event={"ID":"4788f201-3d97-4c2e-ba51-7a38df63b411","Type":"ContainerStarted","Data":"9e706c3abe6f2d2c9c7fd22c45385146da0e366091fca1673a36f0d273df4af9"}
Jul 17 21:59:30.600631 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:30.600443    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9" event={"ID":"4788f201-3d97-4c2e-ba51-7a38df63b411","Type":"ContainerStarted","Data":"eba49c838cdbee325434ae41b334b396687461ad8f8921b9ecd61b5cba7f836e"}
Jul 17 21:59:30.600631 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:30.600587    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"
Jul 17 21:59:30.602304 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:30.602279    2569 generic.go:356] "Generic (PLEG): container finished" podID="3b6d4391-ecd7-4b1b-937b-3763eae81905" containerID="aab21bc6eb955ad2085136281d3d4c166d81f9e55e6eb3006d8559b0d37fec21" exitCode=6
Jul 17 21:59:30.602384 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:30.602356    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj" event={"ID":"3b6d4391-ecd7-4b1b-937b-3763eae81905","Type":"ContainerDied","Data":"aab21bc6eb955ad2085136281d3d4c166d81f9e55e6eb3006d8559b0d37fec21"}
Jul 17 21:59:30.602462 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:30.602421    2569 scope.go:117] "RemoveContainer" containerID="f9914d275170c653555210d12184416e38be14d4b53dede289d14b2592908d17"
Jul 17 21:59:30.602572 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:30.602558    2569 scope.go:117] "RemoveContainer" containerID="aab21bc6eb955ad2085136281d3d4c166d81f9e55e6eb3006d8559b0d37fec21"
Jul 17 21:59:30.602771 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:59:30.602754    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj_odh-ai-gateway-infra(3b6d4391-ecd7-4b1b-937b-3763eae81905)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj" podUID="3b6d4391-ecd7-4b1b-937b-3763eae81905"
Jul 17 21:59:30.622522 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:30.622477    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9" podStartSLOduration=1.315724382 podStartE2EDuration="1.622462532s" totalImagesPullingTime="306.73815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:59:30.620576303 +0000 UTC m=+3084.380142998" watchObservedRunningTime="2026-07-17 21:59:30.622462532 +0000 UTC m=+3084.382029226"
Jul 17 21:59:31.610268 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:31.610237    2569 scope.go:117] "RemoveContainer" containerID="aab21bc6eb955ad2085136281d3d4c166d81f9e55e6eb3006d8559b0d37fec21"
Jul 17 21:59:31.610752 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:59:31.610442    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj_odh-ai-gateway-infra(3b6d4391-ecd7-4b1b-937b-3763eae81905)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj" podUID="3b6d4391-ecd7-4b1b-937b-3763eae81905"
Jul 17 21:59:33.427177 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:33.427146    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:36.615446 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:36.615382    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"
Jul 17 21:59:36.655494 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:36.655460    2569 scope.go:117] "RemoveContainer" containerID="8c86a820f6ca6fc8e06ac12b1625261f8f5c3cec948e063de31d9b35d2fbd5ec"
Jul 17 21:59:37.649937 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:37.649853    2569 generic.go:356] "Generic (PLEG): container finished" podID="4697739a-79e3-41fc-9b47-2e8993968f40" containerID="d4e6ffe05b994b329ff86556d378cf3fc4896891feabe81c21b816f96e336565" exitCode=6
Jul 17 21:59:37.649937 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:37.649880    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8" event={"ID":"4697739a-79e3-41fc-9b47-2e8993968f40","Type":"ContainerDied","Data":"d4e6ffe05b994b329ff86556d378cf3fc4896891feabe81c21b816f96e336565"}
Jul 17 21:59:37.650441 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:37.649942    2569 scope.go:117] "RemoveContainer" containerID="8c86a820f6ca6fc8e06ac12b1625261f8f5c3cec948e063de31d9b35d2fbd5ec"
Jul 17 21:59:37.650441 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:37.650238    2569 scope.go:117] "RemoveContainer" containerID="d4e6ffe05b994b329ff86556d378cf3fc4896891feabe81c21b816f96e336565"
Jul 17 21:59:37.650566 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:59:37.650483    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-vxbm8_odh-ai-gateway-infra(4697739a-79e3-41fc-9b47-2e8993968f40)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8" podUID="4697739a-79e3-41fc-9b47-2e8993968f40"
Jul 17 21:59:38.675875 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:38.675839    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8"]
Jul 17 21:59:38.813801 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:38.813775    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8"
Jul 17 21:59:38.887549 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:38.887512    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4697739a-79e3-41fc-9b47-2e8993968f40-maas-api-service-ca\") pod \"4697739a-79e3-41fc-9b47-2e8993968f40\" (UID: \"4697739a-79e3-41fc-9b47-2e8993968f40\") "
Jul 17 21:59:38.887801 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:38.887780    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4697739a-79e3-41fc-9b47-2e8993968f40-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4697739a-79e3-41fc-9b47-2e8993968f40" (UID: "4697739a-79e3-41fc-9b47-2e8993968f40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:59:38.989015 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:38.988932    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4697739a-79e3-41fc-9b47-2e8993968f40-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:59:39.219064 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:39.219030    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:59:39.672170 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:39.672123    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8" event={"ID":"4697739a-79e3-41fc-9b47-2e8993968f40","Type":"ContainerDied","Data":"9d9a9efbbb3097c0533f3faef9e851570a109eea31dc4509ffd70ec79e90164b"}
Jul 17 21:59:39.672344 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:39.672213    2569 scope.go:117] "RemoveContainer" containerID="d4e6ffe05b994b329ff86556d378cf3fc4896891feabe81c21b816f96e336565"
Jul 17 21:59:39.672387 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:39.672361    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8"
Jul 17 21:59:39.698547 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:39.698517    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8"]
Jul 17 21:59:39.700923 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:39.700899    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-vxbm8"]
Jul 17 21:59:40.657152 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:40.657122    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4697739a-79e3-41fc-9b47-2e8993968f40" path="/var/lib/kubelet/pods/4697739a-79e3-41fc-9b47-2e8993968f40/volumes"
Jul 17 21:59:42.902184 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:42.902151    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:43.003752 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:43.003716    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:45.651159 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:45.651131    2569 scope.go:117] "RemoveContainer" containerID="aab21bc6eb955ad2085136281d3d4c166d81f9e55e6eb3006d8559b0d37fec21"
Jul 17 21:59:46.714690 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:46.714601    2569 generic.go:356] "Generic (PLEG): container finished" podID="3b6d4391-ecd7-4b1b-937b-3763eae81905" containerID="08681a5aab5c5c75396eb7a65cf8975866cbb962f693bb7e2cffda50ce906334" exitCode=6
Jul 17 21:59:46.714690 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:46.714645    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj" event={"ID":"3b6d4391-ecd7-4b1b-937b-3763eae81905","Type":"ContainerDied","Data":"08681a5aab5c5c75396eb7a65cf8975866cbb962f693bb7e2cffda50ce906334"}
Jul 17 21:59:46.715238 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:46.714694    2569 scope.go:117] "RemoveContainer" containerID="aab21bc6eb955ad2085136281d3d4c166d81f9e55e6eb3006d8559b0d37fec21"
Jul 17 21:59:46.715238 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:46.715149    2569 scope.go:117] "RemoveContainer" containerID="08681a5aab5c5c75396eb7a65cf8975866cbb962f693bb7e2cffda50ce906334"
Jul 17 21:59:46.715457 ip-10-0-140-117 kubenswrapper[2569]: E0717 21:59:46.715425    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj_odh-ai-gateway-infra(3b6d4391-ecd7-4b1b-937b-3763eae81905)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj" podUID="3b6d4391-ecd7-4b1b-937b-3763eae81905"
Jul 17 21:59:47.738766 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:47.738729    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj"]
Jul 17 21:59:47.874680 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:47.874653    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj"
Jul 17 21:59:47.971558 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:47.971509    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b6d4391-ecd7-4b1b-937b-3763eae81905-maas-api-service-ca\") pod \"3b6d4391-ecd7-4b1b-937b-3763eae81905\" (UID: \"3b6d4391-ecd7-4b1b-937b-3763eae81905\") "
Jul 17 21:59:47.971907 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:47.971883    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b6d4391-ecd7-4b1b-937b-3763eae81905-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b6d4391-ecd7-4b1b-937b-3763eae81905" (UID: "3b6d4391-ecd7-4b1b-937b-3763eae81905"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:59:48.072273 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:48.072185    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b6d4391-ecd7-4b1b-937b-3763eae81905-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 21:59:48.729578 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:48.729548    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj"
Jul 17 21:59:48.729775 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:48.729588    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj" event={"ID":"3b6d4391-ecd7-4b1b-937b-3763eae81905","Type":"ContainerDied","Data":"52cfa7574b54c0dcf7eb09d53aa0b8475b82c91ce2360635e89de03ba6d6cfa7"}
Jul 17 21:59:48.729775 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:48.729649    2569 scope.go:117] "RemoveContainer" containerID="08681a5aab5c5c75396eb7a65cf8975866cbb962f693bb7e2cffda50ce906334"
Jul 17 21:59:48.746357 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:48.746332    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj"]
Jul 17 21:59:48.752293 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:48.752270    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-jfxjj"]
Jul 17 21:59:50.230272 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.230232    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"]
Jul 17 21:59:50.230850 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.230829    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4697739a-79e3-41fc-9b47-2e8993968f40" containerName="revoke-keys"
Jul 17 21:59:50.230930 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.230853    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4697739a-79e3-41fc-9b47-2e8993968f40" containerName="revoke-keys"
Jul 17 21:59:50.230930 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.230865    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b6d4391-ecd7-4b1b-937b-3763eae81905" containerName="revoke-keys"
Jul 17 21:59:50.230930 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.230873    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b6d4391-ecd7-4b1b-937b-3763eae81905" containerName="revoke-keys"
Jul 17 21:59:50.230930 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.230903    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b6d4391-ecd7-4b1b-937b-3763eae81905" containerName="revoke-keys"
Jul 17 21:59:50.230930 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.230911    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b6d4391-ecd7-4b1b-937b-3763eae81905" containerName="revoke-keys"
Jul 17 21:59:50.231182 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.230936    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4697739a-79e3-41fc-9b47-2e8993968f40" containerName="revoke-keys"
Jul 17 21:59:50.231182 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.230945    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4697739a-79e3-41fc-9b47-2e8993968f40" containerName="revoke-keys"
Jul 17 21:59:50.231182 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.231044    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4697739a-79e3-41fc-9b47-2e8993968f40" containerName="revoke-keys"
Jul 17 21:59:50.231182 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.231059    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b6d4391-ecd7-4b1b-937b-3763eae81905" containerName="revoke-keys"
Jul 17 21:59:50.231182 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.231069    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b6d4391-ecd7-4b1b-937b-3763eae81905" containerName="revoke-keys"
Jul 17 21:59:50.231182 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.231081    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4697739a-79e3-41fc-9b47-2e8993968f40" containerName="revoke-keys"
Jul 17 21:59:50.231182 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.231090    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4697739a-79e3-41fc-9b47-2e8993968f40" containerName="revoke-keys"
Jul 17 21:59:50.235551 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.235525    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 21:59:50.238545 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.238523    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ac333d\"/\"auth-test-model-ac333d-kserve-self-signed-certs\""
Jul 17 21:59:50.239148 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.239132    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ac333d\"/\"openshift-service-ca.crt\""
Jul 17 21:59:50.239262 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.239172    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ac333d\"/\"default-dockercfg-bnws9\""
Jul 17 21:59:50.239262 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.239170    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ac333d\"/\"kube-root-ca.crt\""
Jul 17 21:59:50.256351 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.256323    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"]
Jul 17 21:59:50.274258 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.274233    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:50.289706 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.289676    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-tmp-dir\") pod \"auth-test-model-ac333d-kserve-86cf96f695-5g8pk\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") " pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 21:59:50.289885 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.289725    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-home\") pod \"auth-test-model-ac333d-kserve-86cf96f695-5g8pk\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") " pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 21:59:50.289885 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.289746    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64e6415-abf1-4ce5-8906-d2e06142eda5-tls-certs\") pod \"auth-test-model-ac333d-kserve-86cf96f695-5g8pk\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") " pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 21:59:50.289885 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.289768    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-dshm\") pod \"auth-test-model-ac333d-kserve-86cf96f695-5g8pk\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") " pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 21:59:50.289885 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.289795    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-model-cache\") pod \"auth-test-model-ac333d-kserve-86cf96f695-5g8pk\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") " pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 21:59:50.289885 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.289813    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8r5v\" (UniqueName: \"kubernetes.io/projected/c64e6415-abf1-4ce5-8906-d2e06142eda5-kube-api-access-q8r5v\") pod \"auth-test-model-ac333d-kserve-86cf96f695-5g8pk\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") " pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 21:59:50.391080 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.391043    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-model-cache\") pod \"auth-test-model-ac333d-kserve-86cf96f695-5g8pk\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") " pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 21:59:50.391080 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.391079    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8r5v\" (UniqueName: \"kubernetes.io/projected/c64e6415-abf1-4ce5-8906-d2e06142eda5-kube-api-access-q8r5v\") pod \"auth-test-model-ac333d-kserve-86cf96f695-5g8pk\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") " pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 21:59:50.391318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.391139    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-tmp-dir\") pod \"auth-test-model-ac333d-kserve-86cf96f695-5g8pk\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") " pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 21:59:50.391318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.391168    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-home\") pod \"auth-test-model-ac333d-kserve-86cf96f695-5g8pk\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") " pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 21:59:50.391318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.391186    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64e6415-abf1-4ce5-8906-d2e06142eda5-tls-certs\") pod \"auth-test-model-ac333d-kserve-86cf96f695-5g8pk\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") " pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 21:59:50.391318 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.391215    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-dshm\") pod \"auth-test-model-ac333d-kserve-86cf96f695-5g8pk\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") " pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 21:59:50.391586 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.391565    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-tmp-dir\") pod \"auth-test-model-ac333d-kserve-86cf96f695-5g8pk\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") " pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 21:59:50.391673 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.391646    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-model-cache\") pod \"auth-test-model-ac333d-kserve-86cf96f695-5g8pk\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") " pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 21:59:50.391673 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.391654    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-home\") pod \"auth-test-model-ac333d-kserve-86cf96f695-5g8pk\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") " pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 21:59:50.393580 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.393557    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-dshm\") pod \"auth-test-model-ac333d-kserve-86cf96f695-5g8pk\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") " pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 21:59:50.393841 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.393819    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64e6415-abf1-4ce5-8906-d2e06142eda5-tls-certs\") pod \"auth-test-model-ac333d-kserve-86cf96f695-5g8pk\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") " pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 21:59:50.398970 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.398950    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8r5v\" (UniqueName: \"kubernetes.io/projected/c64e6415-abf1-4ce5-8906-d2e06142eda5-kube-api-access-q8r5v\") pod \"auth-test-model-ac333d-kserve-86cf96f695-5g8pk\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") " pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 21:59:50.544921 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.544831    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 21:59:50.671771 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.671737    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b6d4391-ecd7-4b1b-937b-3763eae81905" path="/var/lib/kubelet/pods/3b6d4391-ecd7-4b1b-937b-3763eae81905/volumes"
Jul 17 21:59:50.689588 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.689551    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"]
Jul 17 21:59:50.691537 ip-10-0-140-117 kubenswrapper[2569]: W0717 21:59:50.691508    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc64e6415_abf1_4ce5_8906_d2e06142eda5.slice/crio-097252ea60f2d0d0ea50502363763c4571a5df63f5cf96d3d0e8019d44875411 WatchSource:0}: Error finding container 097252ea60f2d0d0ea50502363763c4571a5df63f5cf96d3d0e8019d44875411: Status 404 returned error can't find the container with id 097252ea60f2d0d0ea50502363763c4571a5df63f5cf96d3d0e8019d44875411
Jul 17 21:59:50.743800 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.743770    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk" event={"ID":"c64e6415-abf1-4ce5-8906-d2e06142eda5","Type":"ContainerStarted","Data":"097252ea60f2d0d0ea50502363763c4571a5df63f5cf96d3d0e8019d44875411"}
Jul 17 21:59:50.809427 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.809317    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:50.908094 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:50.908058    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 21:59:51.751264 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:51.751230    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk" event={"ID":"c64e6415-abf1-4ce5-8906-d2e06142eda5","Type":"ContainerStarted","Data":"fb56bbf55e801e638ba6198b49a67e2b0a9754a375a3182ba8c3dc1070c362f0"}
Jul 17 21:59:51.769504 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:51.769447    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk" podStartSLOduration=1.769429828 podStartE2EDuration="1.769429828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:59:51.767730211 +0000 UTC m=+3105.527296904" watchObservedRunningTime="2026-07-17 21:59:51.769429828 +0000 UTC m=+3105.528996513"
Jul 17 21:59:53.721080 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:53.721038    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 21:59:54.258455 ip-10-0-140-117 kubenswrapper[2569]: I0717 21:59:54.258417    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:00:00.544972 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:00.544925    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 22:00:00.545434 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:00.544985    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 22:00:00.557784 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:00.557754    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 22:00:00.814668 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:00.814589    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 22:00:38.458270 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:38.458187    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:00:38.826362 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:38.826264    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"]
Jul 17 22:00:38.826739 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:38.826724    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4697739a-79e3-41fc-9b47-2e8993968f40" containerName="revoke-keys"
Jul 17 22:00:38.826784 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:38.826741    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4697739a-79e3-41fc-9b47-2e8993968f40" containerName="revoke-keys"
Jul 17 22:00:38.826784 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:38.826751    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b6d4391-ecd7-4b1b-937b-3763eae81905" containerName="revoke-keys"
Jul 17 22:00:38.826784 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:38.826756    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b6d4391-ecd7-4b1b-937b-3763eae81905" containerName="revoke-keys"
Jul 17 22:00:38.826879 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:38.826823    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b6d4391-ecd7-4b1b-937b-3763eae81905" containerName="revoke-keys"
Jul 17 22:00:38.830124 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:38.830107    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:38.832929 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:38.832908    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-051ea5\"/\"kube-root-ca.crt\""
Jul 17 22:00:38.833065 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:38.832940    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-051ea5\"/\"openshift-service-ca.crt\""
Jul 17 22:00:38.833065 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:38.832989    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-051ea5\"/\"default-dockercfg-fjmpg\""
Jul 17 22:00:38.833065 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:38.832945    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-051ea5\"/\"auth-test-model-051ea5-kserve-self-signed-certs\""
Jul 17 22:00:38.838549 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:38.838523    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"]
Jul 17 22:00:38.927384 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:38.927353    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:00:38.936862 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:38.936829    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-home\") pod \"auth-test-model-051ea5-kserve-5b55dbf499-qfkft\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") " pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:38.937007 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:38.936869    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-dshm\") pod \"auth-test-model-051ea5-kserve-5b55dbf499-qfkft\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") " pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:38.937007 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:38.936888    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-tmp-dir\") pod \"auth-test-model-051ea5-kserve-5b55dbf499-qfkft\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") " pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:38.937007 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:38.936987    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a45a4f8-80e9-47f5-8732-452608139071-tls-certs\") pod \"auth-test-model-051ea5-kserve-5b55dbf499-qfkft\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") " pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:38.937123 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:38.937033    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-model-cache\") pod \"auth-test-model-051ea5-kserve-5b55dbf499-qfkft\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") " pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:38.937123 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:38.937098    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjz4r\" (UniqueName: \"kubernetes.io/projected/5a45a4f8-80e9-47f5-8732-452608139071-kube-api-access-fjz4r\") pod \"auth-test-model-051ea5-kserve-5b55dbf499-qfkft\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") " pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:39.037977 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:39.037933    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjz4r\" (UniqueName: \"kubernetes.io/projected/5a45a4f8-80e9-47f5-8732-452608139071-kube-api-access-fjz4r\") pod \"auth-test-model-051ea5-kserve-5b55dbf499-qfkft\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") " pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:39.038164 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:39.038086    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-home\") pod \"auth-test-model-051ea5-kserve-5b55dbf499-qfkft\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") " pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:39.038164 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:39.038121    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-dshm\") pod \"auth-test-model-051ea5-kserve-5b55dbf499-qfkft\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") " pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:39.038164 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:39.038140    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-tmp-dir\") pod \"auth-test-model-051ea5-kserve-5b55dbf499-qfkft\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") " pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:39.038302 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:39.038187    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a45a4f8-80e9-47f5-8732-452608139071-tls-certs\") pod \"auth-test-model-051ea5-kserve-5b55dbf499-qfkft\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") " pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:39.038302 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:39.038227    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-model-cache\") pod \"auth-test-model-051ea5-kserve-5b55dbf499-qfkft\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") " pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:39.038583 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:39.038560    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-home\") pod \"auth-test-model-051ea5-kserve-5b55dbf499-qfkft\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") " pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:39.038679 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:39.038579    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-model-cache\") pod \"auth-test-model-051ea5-kserve-5b55dbf499-qfkft\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") " pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:39.038679 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:39.038624    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-tmp-dir\") pod \"auth-test-model-051ea5-kserve-5b55dbf499-qfkft\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") " pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:39.040461 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:39.040442    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-dshm\") pod \"auth-test-model-051ea5-kserve-5b55dbf499-qfkft\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") " pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:39.040751 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:39.040731    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a45a4f8-80e9-47f5-8732-452608139071-tls-certs\") pod \"auth-test-model-051ea5-kserve-5b55dbf499-qfkft\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") " pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:39.045417 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:39.045381    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjz4r\" (UniqueName: \"kubernetes.io/projected/5a45a4f8-80e9-47f5-8732-452608139071-kube-api-access-fjz4r\") pod \"auth-test-model-051ea5-kserve-5b55dbf499-qfkft\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") " pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:39.141367 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:39.141317    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:39.281273 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:39.281240    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"]
Jul 17 22:00:39.283575 ip-10-0-140-117 kubenswrapper[2569]: W0717 22:00:39.283546    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a45a4f8_80e9_47f5_8732_452608139071.slice/crio-417202190e6e68225cbc79bec3f913dea461cbe23bacc599fe431c2ab118ca90 WatchSource:0}: Error finding container 417202190e6e68225cbc79bec3f913dea461cbe23bacc599fe431c2ab118ca90: Status 404 returned error can't find the container with id 417202190e6e68225cbc79bec3f913dea461cbe23bacc599fe431c2ab118ca90
Jul 17 22:00:40.033490 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:40.033453    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft" event={"ID":"5a45a4f8-80e9-47f5-8732-452608139071","Type":"ContainerStarted","Data":"433759e0269900a065d56c31cac4998c3ebaa202b209d34959b8b85e0361b0de"}
Jul 17 22:00:40.033490 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:40.033497    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft" event={"ID":"5a45a4f8-80e9-47f5-8732-452608139071","Type":"ContainerStarted","Data":"417202190e6e68225cbc79bec3f913dea461cbe23bacc599fe431c2ab118ca90"}
Jul 17 22:00:40.049790 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:40.049743    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft" podStartSLOduration=2.049729051 podStartE2EDuration="2.049729051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:00:40.049528484 +0000 UTC m=+3153.809095178" watchObservedRunningTime="2026-07-17 22:00:40.049729051 +0000 UTC m=+3153.809295745"
Jul 17 22:00:42.722839 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:42.722804    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 22:00:49.141485 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:49.141427    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:49.141485 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:49.141500    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:49.154179 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:49.154152    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:00:50.101181 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:00:50.101151    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:01:03.320890 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:01:03.320857    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 22:01:19.314980 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:01:19.314944    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 22:01:36.026921 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:01:36.026879    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 22:01:50.423940 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:01:50.423900    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 22:02:07.618524 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:02:07.618441    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 22:02:24.518685 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:02:24.518649    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 22:02:41.113342 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:02:41.113304    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 22:02:45.597206 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:02:45.597173    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"]
Jul 17 22:02:46.351619 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:02:46.351590    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:02:46.694677 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:02:46.694601    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"]
Jul 17 22:02:47.737606 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:02:47.737572    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:02:58.148243 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:02:58.148205    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:02:58.220921 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:02:58.220881    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 22:03:00.617569 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:00.617521    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk" podUID="c64e6415-abf1-4ce5-8906-d2e06142eda5" containerName="main" containerID="cri-o://fb56bbf55e801e638ba6198b49a67e2b0a9754a375a3182ba8c3dc1070c362f0" gracePeriod=45
Jul 17 22:03:00.878364 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:00.878291    2569 generic.go:356] "Generic (PLEG): container finished" podID="c64e6415-abf1-4ce5-8906-d2e06142eda5" containerID="fb56bbf55e801e638ba6198b49a67e2b0a9754a375a3182ba8c3dc1070c362f0" exitCode=0
Jul 17 22:03:00.878540 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:00.878360    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk" event={"ID":"c64e6415-abf1-4ce5-8906-d2e06142eda5","Type":"ContainerDied","Data":"fb56bbf55e801e638ba6198b49a67e2b0a9754a375a3182ba8c3dc1070c362f0"}
Jul 17 22:03:00.878540 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:00.878439    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk" event={"ID":"c64e6415-abf1-4ce5-8906-d2e06142eda5","Type":"ContainerDied","Data":"097252ea60f2d0d0ea50502363763c4571a5df63f5cf96d3d0e8019d44875411"}
Jul 17 22:03:00.878540 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:00.878454    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="097252ea60f2d0d0ea50502363763c4571a5df63f5cf96d3d0e8019d44875411"
Jul 17 22:03:00.880654 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:00.880629    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 22:03:01.027181 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.027150    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-home\") pod \"c64e6415-abf1-4ce5-8906-d2e06142eda5\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") "
Jul 17 22:03:01.027394 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.027196    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-tmp-dir\") pod \"c64e6415-abf1-4ce5-8906-d2e06142eda5\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") "
Jul 17 22:03:01.027394 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.027240    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64e6415-abf1-4ce5-8906-d2e06142eda5-tls-certs\") pod \"c64e6415-abf1-4ce5-8906-d2e06142eda5\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") "
Jul 17 22:03:01.027394 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.027273    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-dshm\") pod \"c64e6415-abf1-4ce5-8906-d2e06142eda5\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") "
Jul 17 22:03:01.027394 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.027318    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8r5v\" (UniqueName: \"kubernetes.io/projected/c64e6415-abf1-4ce5-8906-d2e06142eda5-kube-api-access-q8r5v\") pod \"c64e6415-abf1-4ce5-8906-d2e06142eda5\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") "
Jul 17 22:03:01.027394 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.027360    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-model-cache\") pod \"c64e6415-abf1-4ce5-8906-d2e06142eda5\" (UID: \"c64e6415-abf1-4ce5-8906-d2e06142eda5\") "
Jul 17 22:03:01.027729 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.027529    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c64e6415-abf1-4ce5-8906-d2e06142eda5" (UID: "c64e6415-abf1-4ce5-8906-d2e06142eda5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:03:01.027729 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.027540    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-home" (OuterVolumeSpecName: "home") pod "c64e6415-abf1-4ce5-8906-d2e06142eda5" (UID: "c64e6415-abf1-4ce5-8906-d2e06142eda5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:03:01.027729 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.027627    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-home\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:03:01.027729 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.027679    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-tmp-dir\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:03:01.027865 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.027800    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-model-cache" (OuterVolumeSpecName: "model-cache") pod "c64e6415-abf1-4ce5-8906-d2e06142eda5" (UID: "c64e6415-abf1-4ce5-8906-d2e06142eda5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:03:01.029695 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.029665    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-dshm" (OuterVolumeSpecName: "dshm") pod "c64e6415-abf1-4ce5-8906-d2e06142eda5" (UID: "c64e6415-abf1-4ce5-8906-d2e06142eda5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:03:01.029793 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.029710    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64e6415-abf1-4ce5-8906-d2e06142eda5-kube-api-access-q8r5v" (OuterVolumeSpecName: "kube-api-access-q8r5v") pod "c64e6415-abf1-4ce5-8906-d2e06142eda5" (UID: "c64e6415-abf1-4ce5-8906-d2e06142eda5"). InnerVolumeSpecName "kube-api-access-q8r5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:03:01.029793 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.029736    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64e6415-abf1-4ce5-8906-d2e06142eda5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c64e6415-abf1-4ce5-8906-d2e06142eda5" (UID: "c64e6415-abf1-4ce5-8906-d2e06142eda5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 22:03:01.129028 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.128944    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-model-cache\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:03:01.129028 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.128976    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64e6415-abf1-4ce5-8906-d2e06142eda5-tls-certs\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:03:01.129028 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.128984    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64e6415-abf1-4ce5-8906-d2e06142eda5-dshm\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:03:01.129028 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.128993    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8r5v\" (UniqueName: \"kubernetes.io/projected/c64e6415-abf1-4ce5-8906-d2e06142eda5-kube-api-access-q8r5v\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:03:01.717652 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.717604    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft" podUID="5a45a4f8-80e9-47f5-8732-452608139071" containerName="main" containerID="cri-o://433759e0269900a065d56c31cac4998c3ebaa202b209d34959b8b85e0361b0de" gracePeriod=45
Jul 17 22:03:01.888414 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.888365    2569 generic.go:356] "Generic (PLEG): container finished" podID="5a45a4f8-80e9-47f5-8732-452608139071" containerID="433759e0269900a065d56c31cac4998c3ebaa202b209d34959b8b85e0361b0de" exitCode=0
Jul 17 22:03:01.888594 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.888427    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft" event={"ID":"5a45a4f8-80e9-47f5-8732-452608139071","Type":"ContainerDied","Data":"433759e0269900a065d56c31cac4998c3ebaa202b209d34959b8b85e0361b0de"}
Jul 17 22:03:01.888594 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.888505    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"
Jul 17 22:03:01.912845 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.912708    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"]
Jul 17 22:03:01.915724 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.915690    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk"]
Jul 17 22:03:01.985792 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:01.985767    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:03:02.138705 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.138672    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-tmp-dir\") pod \"5a45a4f8-80e9-47f5-8732-452608139071\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") "
Jul 17 22:03:02.138914 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.138718    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-model-cache\") pod \"5a45a4f8-80e9-47f5-8732-452608139071\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") "
Jul 17 22:03:02.138914 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.138746    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a45a4f8-80e9-47f5-8732-452608139071-tls-certs\") pod \"5a45a4f8-80e9-47f5-8732-452608139071\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") "
Jul 17 22:03:02.138914 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.138812    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-dshm\") pod \"5a45a4f8-80e9-47f5-8732-452608139071\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") "
Jul 17 22:03:02.138914 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.138840    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjz4r\" (UniqueName: \"kubernetes.io/projected/5a45a4f8-80e9-47f5-8732-452608139071-kube-api-access-fjz4r\") pod \"5a45a4f8-80e9-47f5-8732-452608139071\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") "
Jul 17 22:03:02.138914 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.138881    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-home\") pod \"5a45a4f8-80e9-47f5-8732-452608139071\" (UID: \"5a45a4f8-80e9-47f5-8732-452608139071\") "
Jul 17 22:03:02.139181 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.139032    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a45a4f8-80e9-47f5-8732-452608139071" (UID: "5a45a4f8-80e9-47f5-8732-452608139071"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:03:02.139181 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.139044    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a45a4f8-80e9-47f5-8732-452608139071" (UID: "5a45a4f8-80e9-47f5-8732-452608139071"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:03:02.139292 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.139179    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-home" (OuterVolumeSpecName: "home") pod "5a45a4f8-80e9-47f5-8732-452608139071" (UID: "5a45a4f8-80e9-47f5-8732-452608139071"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:03:02.139292 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.139207    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-tmp-dir\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:03:02.139292 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.139228    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-model-cache\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:03:02.141157 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.141137    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a45a4f8-80e9-47f5-8732-452608139071-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a45a4f8-80e9-47f5-8732-452608139071" (UID: "5a45a4f8-80e9-47f5-8732-452608139071"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 22:03:02.141265 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.141173    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-dshm" (OuterVolumeSpecName: "dshm") pod "5a45a4f8-80e9-47f5-8732-452608139071" (UID: "5a45a4f8-80e9-47f5-8732-452608139071"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:03:02.141265 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.141206    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a45a4f8-80e9-47f5-8732-452608139071-kube-api-access-fjz4r" (OuterVolumeSpecName: "kube-api-access-fjz4r") pod "5a45a4f8-80e9-47f5-8732-452608139071" (UID: "5a45a4f8-80e9-47f5-8732-452608139071"). InnerVolumeSpecName "kube-api-access-fjz4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:03:02.239953 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.239865    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-dshm\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:03:02.239953 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.239900    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjz4r\" (UniqueName: \"kubernetes.io/projected/5a45a4f8-80e9-47f5-8732-452608139071-kube-api-access-fjz4r\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:03:02.239953 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.239909    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a45a4f8-80e9-47f5-8732-452608139071-home\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:03:02.239953 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.239919    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a45a4f8-80e9-47f5-8732-452608139071-tls-certs\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:03:02.657420 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.657366    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c64e6415-abf1-4ce5-8906-d2e06142eda5" path="/var/lib/kubelet/pods/c64e6415-abf1-4ce5-8906-d2e06142eda5/volumes"
Jul 17 22:03:02.895743 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.895706    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"
Jul 17 22:03:02.895743 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.895727    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft" event={"ID":"5a45a4f8-80e9-47f5-8732-452608139071","Type":"ContainerDied","Data":"417202190e6e68225cbc79bec3f913dea461cbe23bacc599fe431c2ab118ca90"}
Jul 17 22:03:02.896239 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.895778    2569 scope.go:117] "RemoveContainer" containerID="433759e0269900a065d56c31cac4998c3ebaa202b209d34959b8b85e0361b0de"
Jul 17 22:03:02.911963 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.911876    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"]
Jul 17 22:03:02.914985 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:02.914954    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-051ea5/auth-test-model-051ea5-kserve-5b55dbf499-qfkft"]
Jul 17 22:03:04.657829 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:04.657790    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a45a4f8-80e9-47f5-8732-452608139071" path="/var/lib/kubelet/pods/5a45a4f8-80e9-47f5-8732-452608139071/volumes"
Jul 17 22:03:05.803884 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:05.803841    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-ac333d/auth-test-model-ac333d-kserve-86cf96f695-5g8pk" podUID="c64e6415-abf1-4ce5-8906-d2e06142eda5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 17 22:03:06.946314 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:06.946287    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 22:03:06.980128 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:06.980096    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 22:03:07.954613 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:07.954575    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:03:08.056081 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:08.056046    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:03:10.817235 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:10.817202    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 22:03:40.969036 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:40.968952    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:03:43.755894 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.755859    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"]
Jul 17 22:03:43.756730 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.756709    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c64e6415-abf1-4ce5-8906-d2e06142eda5" containerName="main"
Jul 17 22:03:43.756730 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.756732    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c64e6415-abf1-4ce5-8906-d2e06142eda5" containerName="main"
Jul 17 22:03:43.756897 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.756751    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a45a4f8-80e9-47f5-8732-452608139071" containerName="main"
Jul 17 22:03:43.756897 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.756759    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a45a4f8-80e9-47f5-8732-452608139071" containerName="main"
Jul 17 22:03:43.756897 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.756867    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c64e6415-abf1-4ce5-8906-d2e06142eda5" containerName="main"
Jul 17 22:03:43.756897 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.756878    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a45a4f8-80e9-47f5-8732-452608139071" containerName="main"
Jul 17 22:03:43.760239 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.760210    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:43.762973 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.762926    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-051ea5\"/\"sub-test-model-051ea5-kserve-self-signed-certs\""
Jul 17 22:03:43.763123 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.762969    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-051ea5\"/\"openshift-service-ca.crt\""
Jul 17 22:03:43.763123 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.763007    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-051ea5\"/\"kube-root-ca.crt\""
Jul 17 22:03:43.763251 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.763214    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-051ea5\"/\"default-dockercfg-fjmpg\""
Jul 17 22:03:43.769711 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.769687    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"]
Jul 17 22:03:43.790072 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.790031    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:03:43.806143 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.806110    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-model-cache\") pod \"sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") " pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:43.806143 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.806144    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-tmp-dir\") pod \"sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") " pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:43.806317 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.806163    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-home\") pod \"sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") " pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:43.806317 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.806218    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-dshm\") pod \"sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") " pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:43.806317 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.806295    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-tls-certs\") pod \"sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") " pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:43.806461 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.806354    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2422\" (UniqueName: \"kubernetes.io/projected/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-kube-api-access-t2422\") pod \"sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") " pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:43.907360 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.907327    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-tls-certs\") pod \"sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") " pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:43.907554 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.907379    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2422\" (UniqueName: \"kubernetes.io/projected/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-kube-api-access-t2422\") pod \"sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") " pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:43.907614 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.907567    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-model-cache\") pod \"sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") " pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:43.907614 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.907606    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-tmp-dir\") pod \"sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") " pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:43.907709 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.907638    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-home\") pod \"sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") " pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:43.907709 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.907681    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-dshm\") pod \"sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") " pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:43.907992 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.907970    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-model-cache\") pod \"sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") " pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:43.908068 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.908045    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-home\") pod \"sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") " pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:43.908129 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.908070    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-tmp-dir\") pod \"sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") " pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:43.909825 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.909802    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-dshm\") pod \"sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") " pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:43.910008 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.909992    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-tls-certs\") pod \"sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") " pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:43.914728 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:43.914708    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2422\" (UniqueName: \"kubernetes.io/projected/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-kube-api-access-t2422\") pod \"sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") " pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:44.071713 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:44.071628    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:44.207608 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:44.207581    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"]
Jul 17 22:03:44.208938 ip-10-0-140-117 kubenswrapper[2569]: W0717 22:03:44.208906    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88f19508_b2f3_4691_8cc6_b283d4c5f9e5.slice/crio-7b77ea38dfe1cddfd6528f789ccf71a7785f7d94d1d33b1f4b081e8bb398dad0 WatchSource:0}: Error finding container 7b77ea38dfe1cddfd6528f789ccf71a7785f7d94d1d33b1f4b081e8bb398dad0: Status 404 returned error can't find the container with id 7b77ea38dfe1cddfd6528f789ccf71a7785f7d94d1d33b1f4b081e8bb398dad0
Jul 17 22:03:45.140477 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:45.140437    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv" event={"ID":"88f19508-b2f3-4691-8cc6-b283d4c5f9e5","Type":"ContainerStarted","Data":"ef693637751d9a87672d91a665bca912f8e5da7ca99c2a19d729d8813e517224"}
Jul 17 22:03:45.140477 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:45.140482    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv" event={"ID":"88f19508-b2f3-4691-8cc6-b283d4c5f9e5","Type":"ContainerStarted","Data":"7b77ea38dfe1cddfd6528f789ccf71a7785f7d94d1d33b1f4b081e8bb398dad0"}
Jul 17 22:03:45.160650 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:45.160590    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv" podStartSLOduration=2.160569644 podStartE2EDuration="2.160569644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:03:45.155758002 +0000 UTC m=+3338.915324697" watchObservedRunningTime="2026-07-17 22:03:45.160569644 +0000 UTC m=+3338.920136340"
Jul 17 22:03:47.391159 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:47.391128    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:03:47.443573 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:47.443546    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:03:47.915733 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:47.915697    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 22:03:54.072593 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:54.072559    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:54.072593 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:54.072602    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:54.085187 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:54.085160    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:54.204068 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:54.204038    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:03:59.357814 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:59.357777    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:03:59.456265 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:03:59.456229    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:04:03.994852 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:03.994815    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:04:04.096665 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:04.096627    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:04:18.622012 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:18.621972    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 22:04:37.015269 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:37.015232    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 22:04:43.426552 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:43.426517    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:04:43.607862 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:43.607818    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:04:43.710008 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:43.709920    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:04:44.406871 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:44.406838    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"]
Jul 17 22:04:44.424985 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:44.424950    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:04:51.746928 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:51.746895    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:04:51.847410 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:51.847369    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:04:53.623253 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.623220    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"]
Jul 17 22:04:53.627429 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.627395    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:04:53.629825 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.629802    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ac333d\"/\"openshift-service-ca.crt\""
Jul 17 22:04:53.629934 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.629835    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ac333d\"/\"default-dockercfg-bnws9\""
Jul 17 22:04:53.629934 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.629835    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ac333d\"/\"kube-root-ca.crt\""
Jul 17 22:04:53.630085 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.629802    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ac333d\"/\"rate-test-model-ac333d-kserve-self-signed-certs\""
Jul 17 22:04:53.634621 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.634597    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"]
Jul 17 22:04:53.646244 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.646213    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:04:53.747414 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.747379    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-dshm\") pod \"rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") " pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:04:53.747617 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.747442    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c896984-571c-42bf-819a-10e3cb6ee2f7-tls-certs\") pod \"rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") " pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:04:53.747617 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.747476    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-model-cache\") pod \"rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") " pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:04:53.747617 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.747580    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swkv8\" (UniqueName: \"kubernetes.io/projected/0c896984-571c-42bf-819a-10e3cb6ee2f7-kube-api-access-swkv8\") pod \"rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") " pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:04:53.747617 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.747614    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-home\") pod \"rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") " pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:04:53.747811 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.747661    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-tmp-dir\") pod \"rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") " pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:04:53.848642 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.848609    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swkv8\" (UniqueName: \"kubernetes.io/projected/0c896984-571c-42bf-819a-10e3cb6ee2f7-kube-api-access-swkv8\") pod \"rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") " pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:04:53.848830 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.848649    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-home\") pod \"rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") " pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:04:53.848830 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.848677    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-tmp-dir\") pod \"rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") " pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:04:53.848830 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.848713    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-dshm\") pod \"rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") " pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:04:53.848830 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.848727    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c896984-571c-42bf-819a-10e3cb6ee2f7-tls-certs\") pod \"rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") " pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:04:53.848830 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.848746    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-model-cache\") pod \"rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") " pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:04:53.849104 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.849082    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-home\") pod \"rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") " pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:04:53.849150 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.849104    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-model-cache\") pod \"rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") " pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:04:53.849187 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.849157    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-tmp-dir\") pod \"rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") " pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:04:53.851052 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.851030    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-dshm\") pod \"rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") " pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:04:53.851171 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.851154    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c896984-571c-42bf-819a-10e3cb6ee2f7-tls-certs\") pod \"rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") " pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:04:53.856708 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.856670    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swkv8\" (UniqueName: \"kubernetes.io/projected/0c896984-571c-42bf-819a-10e3cb6ee2f7-kube-api-access-swkv8\") pod \"rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") " pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:04:53.939058 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:53.938990    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:04:54.075869 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:54.075834    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"]
Jul 17 22:04:54.076469 ip-10-0-140-117 kubenswrapper[2569]: W0717 22:04:54.076441    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c896984_571c_42bf_819a_10e3cb6ee2f7.slice/crio-5d663f224c16417f0482f9c1345d518aac1456a6c70cf4772ebc055a4df112bc WatchSource:0}: Error finding container 5d663f224c16417f0482f9c1345d518aac1456a6c70cf4772ebc055a4df112bc: Status 404 returned error can't find the container with id 5d663f224c16417f0482f9c1345d518aac1456a6c70cf4772ebc055a4df112bc
Jul 17 22:04:54.546602 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:54.546560    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc" event={"ID":"0c896984-571c-42bf-819a-10e3cb6ee2f7","Type":"ContainerStarted","Data":"b394f4b7b7fd35c2b6d73c724223d2b2e12f8d402aa68d8f95b2133036e58381"}
Jul 17 22:04:54.546781 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:54.546613    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc" event={"ID":"0c896984-571c-42bf-819a-10e3cb6ee2f7","Type":"ContainerStarted","Data":"5d663f224c16417f0482f9c1345d518aac1456a6c70cf4772ebc055a4df112bc"}
Jul 17 22:04:54.565625 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:54.565574    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc" podStartSLOduration=1.5655592170000001 podStartE2EDuration="1.565559217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:04:54.563939124 +0000 UTC m=+3408.323505846" watchObservedRunningTime="2026-07-17 22:04:54.565559217 +0000 UTC m=+3408.325125911"
Jul 17 22:04:57.735209 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:57.735176    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 22:04:59.427432 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.427330    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv" podUID="88f19508-b2f3-4691-8cc6-b283d4c5f9e5" containerName="main" containerID="cri-o://ef693637751d9a87672d91a665bca912f8e5da7ca99c2a19d729d8813e517224" gracePeriod=45
Jul 17 22:04:59.580678 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.580646    2569 generic.go:356] "Generic (PLEG): container finished" podID="88f19508-b2f3-4691-8cc6-b283d4c5f9e5" containerID="ef693637751d9a87672d91a665bca912f8e5da7ca99c2a19d729d8813e517224" exitCode=0
Jul 17 22:04:59.580847 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.580720    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv" event={"ID":"88f19508-b2f3-4691-8cc6-b283d4c5f9e5","Type":"ContainerDied","Data":"ef693637751d9a87672d91a665bca912f8e5da7ca99c2a19d729d8813e517224"}
Jul 17 22:04:59.688930 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.688872    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:04:59.700142 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.700123    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-home\") pod \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") "
Jul 17 22:04:59.700220 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.700151    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-dshm\") pod \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") "
Jul 17 22:04:59.700220 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.700173    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-model-cache\") pod \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") "
Jul 17 22:04:59.700220 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.700211    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-tls-certs\") pod \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") "
Jul 17 22:04:59.700326 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.700236    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2422\" (UniqueName: \"kubernetes.io/projected/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-kube-api-access-t2422\") pod \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") "
Jul 17 22:04:59.700326 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.700278    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-tmp-dir\") pod \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\" (UID: \"88f19508-b2f3-4691-8cc6-b283d4c5f9e5\") "
Jul 17 22:04:59.700452 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.700392    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-home" (OuterVolumeSpecName: "home") pod "88f19508-b2f3-4691-8cc6-b283d4c5f9e5" (UID: "88f19508-b2f3-4691-8cc6-b283d4c5f9e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:04:59.700513 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.700455    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "88f19508-b2f3-4691-8cc6-b283d4c5f9e5" (UID: "88f19508-b2f3-4691-8cc6-b283d4c5f9e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:04:59.700708 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.700650    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88f19508-b2f3-4691-8cc6-b283d4c5f9e5" (UID: "88f19508-b2f3-4691-8cc6-b283d4c5f9e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:04:59.700708 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.700676    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-home\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:04:59.700861 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.700717    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-model-cache\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:04:59.702292 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.702274    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-dshm" (OuterVolumeSpecName: "dshm") pod "88f19508-b2f3-4691-8cc6-b283d4c5f9e5" (UID: "88f19508-b2f3-4691-8cc6-b283d4c5f9e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:04:59.702765 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.702728    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-kube-api-access-t2422" (OuterVolumeSpecName: "kube-api-access-t2422") pod "88f19508-b2f3-4691-8cc6-b283d4c5f9e5" (UID: "88f19508-b2f3-4691-8cc6-b283d4c5f9e5"). InnerVolumeSpecName "kube-api-access-t2422". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:04:59.703186 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.702814    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88f19508-b2f3-4691-8cc6-b283d4c5f9e5" (UID: "88f19508-b2f3-4691-8cc6-b283d4c5f9e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 22:04:59.801290 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.801253    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2422\" (UniqueName: \"kubernetes.io/projected/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-kube-api-access-t2422\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:04:59.801290 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.801280    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-tmp-dir\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:04:59.801290 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.801291    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-dshm\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:04:59.801290 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:04:59.801299    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f19508-b2f3-4691-8cc6-b283d4c5f9e5-tls-certs\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:05:00.588862 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:00.588828    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"
Jul 17 22:05:00.589314 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:00.588824    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv" event={"ID":"88f19508-b2f3-4691-8cc6-b283d4c5f9e5","Type":"ContainerDied","Data":"7b77ea38dfe1cddfd6528f789ccf71a7785f7d94d1d33b1f4b081e8bb398dad0"}
Jul 17 22:05:00.589314 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:00.588949    2569 scope.go:117] "RemoveContainer" containerID="ef693637751d9a87672d91a665bca912f8e5da7ca99c2a19d729d8813e517224"
Jul 17 22:05:00.610003 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:00.609979    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"]
Jul 17 22:05:00.612513 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:00.612493    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-051ea5/sub-test-model-051ea5-kserve-6dc6bfdbb9-7nmtv"]
Jul 17 22:05:00.663979 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:00.663937    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88f19508-b2f3-4691-8cc6-b283d4c5f9e5" path="/var/lib/kubelet/pods/88f19508-b2f3-4691-8cc6-b283d4c5f9e5/volumes"
Jul 17 22:05:03.939664 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:03.939613    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:05:03.939664 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:03.939675    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:05:03.952670 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:03.952641    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:05:04.624371 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:04.624339    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:05:07.844633 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:07.844564    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:08.111970 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:08.111887    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:08.256214 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:08.256186    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:08.397242 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:08.397127    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:08.454929 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:08.454891    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:08.815689 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:08.815587    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:09.245230 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:09.245197    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:09.342795 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:09.342755    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:10.006882 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:10.006817    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:10.472283 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:10.472249    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:10.570123 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:10.570075    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:10.646670 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:10.646636    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:10.715625 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:10.715590    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:10.821435 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:10.821319    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:11.215313 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:11.215273    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:11.311287 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:11.311248    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:11.599388 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:11.599354    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:11.704035 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:11.703999    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:11.803674 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:11.803631    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:12.393804 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:12.393767    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:12.493465 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:12.493428    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:12.795133 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:12.795059    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:12.872928 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:12.872896    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:39.116177 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.116125    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"]
Jul 17 22:05:39.116874 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.116850    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88f19508-b2f3-4691-8cc6-b283d4c5f9e5" containerName="main"
Jul 17 22:05:39.116967 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.116877    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="88f19508-b2f3-4691-8cc6-b283d4c5f9e5" containerName="main"
Jul 17 22:05:39.117025 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.117012    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="88f19508-b2f3-4691-8cc6-b283d4c5f9e5" containerName="main"
Jul 17 22:05:39.120152 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.120130    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:39.122671 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.122648    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-051ea5\"/\"openshift-service-ca.crt\""
Jul 17 22:05:39.122793 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.122648    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-051ea5\"/\"rate-test-model-051ea5-kserve-self-signed-certs\""
Jul 17 22:05:39.122793 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.122708    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-051ea5\"/\"default-dockercfg-fjmpg\""
Jul 17 22:05:39.122793 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.122756    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-051ea5\"/\"kube-root-ca.crt\""
Jul 17 22:05:39.127568 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.127539    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"]
Jul 17 22:05:39.158509 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.158467    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:05:39.264727 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.264689    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-model-cache\") pod \"rate-test-model-051ea5-kserve-67c47dcd87-sf98p\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") " pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:39.264727 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.264730    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-home\") pod \"rate-test-model-051ea5-kserve-67c47dcd87-sf98p\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") " pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:39.264961 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.264762    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h62wj\" (UniqueName: \"kubernetes.io/projected/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-kube-api-access-h62wj\") pod \"rate-test-model-051ea5-kserve-67c47dcd87-sf98p\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") " pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:39.264961 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.264836    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-tmp-dir\") pod \"rate-test-model-051ea5-kserve-67c47dcd87-sf98p\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") " pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:39.264961 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.264883    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-dshm\") pod \"rate-test-model-051ea5-kserve-67c47dcd87-sf98p\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") " pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:39.265072 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.264966    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-tls-certs\") pod \"rate-test-model-051ea5-kserve-67c47dcd87-sf98p\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") " pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:39.365970 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.365929    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-model-cache\") pod \"rate-test-model-051ea5-kserve-67c47dcd87-sf98p\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") " pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:39.366166 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.365980    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-home\") pod \"rate-test-model-051ea5-kserve-67c47dcd87-sf98p\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") " pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:39.366166 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.366107    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h62wj\" (UniqueName: \"kubernetes.io/projected/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-kube-api-access-h62wj\") pod \"rate-test-model-051ea5-kserve-67c47dcd87-sf98p\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") " pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:39.366166 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.366155    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-tmp-dir\") pod \"rate-test-model-051ea5-kserve-67c47dcd87-sf98p\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") " pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:39.366376 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.366197    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-dshm\") pod \"rate-test-model-051ea5-kserve-67c47dcd87-sf98p\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") " pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:39.366376 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.366285    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-tls-certs\") pod \"rate-test-model-051ea5-kserve-67c47dcd87-sf98p\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") " pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:39.366538 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.366390    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-home\") pod \"rate-test-model-051ea5-kserve-67c47dcd87-sf98p\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") " pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:39.366538 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.366449    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-model-cache\") pod \"rate-test-model-051ea5-kserve-67c47dcd87-sf98p\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") " pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:39.366608 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.366556    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-tmp-dir\") pod \"rate-test-model-051ea5-kserve-67c47dcd87-sf98p\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") " pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:39.368550 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.368526    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-dshm\") pod \"rate-test-model-051ea5-kserve-67c47dcd87-sf98p\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") " pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:39.368789 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.368770    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-tls-certs\") pod \"rate-test-model-051ea5-kserve-67c47dcd87-sf98p\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") " pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:39.373959 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.373934    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h62wj\" (UniqueName: \"kubernetes.io/projected/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-kube-api-access-h62wj\") pod \"rate-test-model-051ea5-kserve-67c47dcd87-sf98p\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") " pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:39.431689 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.431649    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:39.573520 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.573477    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"]
Jul 17 22:05:39.574714 ip-10-0-140-117 kubenswrapper[2569]: W0717 22:05:39.574688    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57cbb9fb_ea20_44e9_af6f_dd59f4d7fa9d.slice/crio-fc67b3b757f059e81744412095219fdc75f61e1e6d293a2f94fd3ce0dfa7b908 WatchSource:0}: Error finding container fc67b3b757f059e81744412095219fdc75f61e1e6d293a2f94fd3ce0dfa7b908: Status 404 returned error can't find the container with id fc67b3b757f059e81744412095219fdc75f61e1e6d293a2f94fd3ce0dfa7b908
Jul 17 22:05:39.820306 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.820272    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p" event={"ID":"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d","Type":"ContainerStarted","Data":"d7b68076fe86a0fe2ea5dc6ab32ab0ee78d7f0d61f36e9c5a549432d25714d81"}
Jul 17 22:05:39.820306 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.820314    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p" event={"ID":"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d","Type":"ContainerStarted","Data":"fc67b3b757f059e81744412095219fdc75f61e1e6d293a2f94fd3ce0dfa7b908"}
Jul 17 22:05:39.835698 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:39.835641    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p" podStartSLOduration=0.835627264 podStartE2EDuration="835.627264ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:05:39.834551233 +0000 UTC m=+3453.594117956" watchObservedRunningTime="2026-07-17 22:05:39.835627264 +0000 UTC m=+3453.595193986"
Jul 17 22:05:44.028775 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:44.028739    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 22:05:49.432454 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:49.432415    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:49.432976 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:49.432477    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:49.445474 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:49.445444    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:49.893117 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:49.893089    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:05:58.621937 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:05:58.621886    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 22:06:17.001823 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:17.001788    2569 scope.go:117] "RemoveContainer" containerID="fb56bbf55e801e638ba6198b49a67e2b0a9754a375a3182ba8c3dc1070c362f0"
Jul 17 22:06:19.112768 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:19.112732    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 22:06:26.894763 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:26.894727    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"]
Jul 17 22:06:26.914467 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:26.914433    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:06:27.824560 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:27.824524    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"]
Jul 17 22:06:27.844278 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:27.844246    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:06:28.759912 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:28.759875    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7257-139628158996784"]
Jul 17 22:06:28.763463 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:28.763441    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158996784"
Jul 17 22:06:28.766333 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:28.766308    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rxnlj\""
Jul 17 22:06:28.771765 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:28.771672    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-139628158996784"]
Jul 17 22:06:28.918430 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:28.918369    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5k4b\" (UniqueName: \"kubernetes.io/projected/0a67f2ee-0b71-4b40-9c13-664296f03a2b-kube-api-access-b5k4b\") pod \"test-curl-7257-139628158996784\" (UID: \"0a67f2ee-0b71-4b40-9c13-664296f03a2b\") " pod="opendatahub/test-curl-7257-139628158996784"
Jul 17 22:06:29.019140 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:29.019044    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5k4b\" (UniqueName: \"kubernetes.io/projected/0a67f2ee-0b71-4b40-9c13-664296f03a2b-kube-api-access-b5k4b\") pod \"test-curl-7257-139628158996784\" (UID: \"0a67f2ee-0b71-4b40-9c13-664296f03a2b\") " pod="opendatahub/test-curl-7257-139628158996784"
Jul 17 22:06:29.027492 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:29.027465    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5k4b\" (UniqueName: \"kubernetes.io/projected/0a67f2ee-0b71-4b40-9c13-664296f03a2b-kube-api-access-b5k4b\") pod \"test-curl-7257-139628158996784\" (UID: \"0a67f2ee-0b71-4b40-9c13-664296f03a2b\") " pod="opendatahub/test-curl-7257-139628158996784"
Jul 17 22:06:29.074569 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:29.074535    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158996784"
Jul 17 22:06:29.213595 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:29.213563    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-139628158996784"]
Jul 17 22:06:29.214814 ip-10-0-140-117 kubenswrapper[2569]: W0717 22:06:29.214776    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a67f2ee_0b71_4b40_9c13_664296f03a2b.slice/crio-289b4047a8003706da2c915b5738ee709a1293772ebfad1e05811607f53efd8c WatchSource:0}: Error finding container 289b4047a8003706da2c915b5738ee709a1293772ebfad1e05811607f53efd8c: Status 404 returned error can't find the container with id 289b4047a8003706da2c915b5738ee709a1293772ebfad1e05811607f53efd8c
Jul 17 22:06:29.216138 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:29.216117    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 22:06:30.132439 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:30.132391    2569 generic.go:356] "Generic (PLEG): container finished" podID="0a67f2ee-0b71-4b40-9c13-664296f03a2b" containerID="3841cea89c2e570f5d1bce721748595e6a32b4ab04fdf8c1ae2b333a82bf2d66" exitCode=0
Jul 17 22:06:30.132821 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:30.132479    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-139628158996784" event={"ID":"0a67f2ee-0b71-4b40-9c13-664296f03a2b","Type":"ContainerDied","Data":"3841cea89c2e570f5d1bce721748595e6a32b4ab04fdf8c1ae2b333a82bf2d66"}
Jul 17 22:06:30.132821 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:30.132523    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-139628158996784" event={"ID":"0a67f2ee-0b71-4b40-9c13-664296f03a2b","Type":"ContainerStarted","Data":"289b4047a8003706da2c915b5738ee709a1293772ebfad1e05811607f53efd8c"}
Jul 17 22:06:31.284272 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:31.284249    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158996784"
Jul 17 22:06:31.302247 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:31.302211    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7257-139628158996784_0a67f2ee-0b71-4b40-9c13-664296f03a2b/test-curl-7257-139628158996784/0.log"
Jul 17 22:06:31.323121 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:31.323088    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7257-139628158996784"]
Jul 17 22:06:31.327359 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:31.327331    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7257-139628158996784"]
Jul 17 22:06:31.441522 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:31.441427    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5k4b\" (UniqueName: \"kubernetes.io/projected/0a67f2ee-0b71-4b40-9c13-664296f03a2b-kube-api-access-b5k4b\") pod \"0a67f2ee-0b71-4b40-9c13-664296f03a2b\" (UID: \"0a67f2ee-0b71-4b40-9c13-664296f03a2b\") "
Jul 17 22:06:31.443659 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:31.443620    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a67f2ee-0b71-4b40-9c13-664296f03a2b-kube-api-access-b5k4b" (OuterVolumeSpecName: "kube-api-access-b5k4b") pod "0a67f2ee-0b71-4b40-9c13-664296f03a2b" (UID: "0a67f2ee-0b71-4b40-9c13-664296f03a2b"). InnerVolumeSpecName "kube-api-access-b5k4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:06:31.542982 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:31.542947    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5k4b\" (UniqueName: \"kubernetes.io/projected/0a67f2ee-0b71-4b40-9c13-664296f03a2b-kube-api-access-b5k4b\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:06:32.145017 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:32.144990    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="289b4047a8003706da2c915b5738ee709a1293772ebfad1e05811607f53efd8c"
Jul 17 22:06:32.145017 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:32.145008    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158996784"
Jul 17 22:06:32.154608 ip-10-0-140-117 kubenswrapper[2569]: E0717 22:06:32.154574    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7257-139628158996784\" is forbidden: User \"system:node:ip-10-0-140-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-117.ec2.internal' and this object" podUID="0a67f2ee-0b71-4b40-9c13-664296f03a2b" pod="opendatahub/test-curl-7257-139628158996784"
Jul 17 22:06:32.657388 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:32.657346    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a67f2ee-0b71-4b40-9c13-664296f03a2b" path="/var/lib/kubelet/pods/0a67f2ee-0b71-4b40-9c13-664296f03a2b/volumes"
Jul 17 22:06:33.515617 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:33.515583    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 22:06:35.019701 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:35.019667    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7257-139628158805312"]
Jul 17 22:06:35.020064 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:35.020051    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a67f2ee-0b71-4b40-9c13-664296f03a2b" containerName="test-curl-7257-139628158996784"
Jul 17 22:06:35.020120 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:35.020066    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a67f2ee-0b71-4b40-9c13-664296f03a2b" containerName="test-curl-7257-139628158996784"
Jul 17 22:06:35.020161 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:35.020147    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a67f2ee-0b71-4b40-9c13-664296f03a2b" containerName="test-curl-7257-139628158996784"
Jul 17 22:06:35.023529 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:35.023506    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158805312"
Jul 17 22:06:35.025378 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:35.025350    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rxnlj\""
Jul 17 22:06:35.026805 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:35.026785    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-139628158805312"]
Jul 17 22:06:35.073969 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:35.073941    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jphl4\" (UniqueName: \"kubernetes.io/projected/73f797d0-65af-45ae-81b2-59db31ac17a7-kube-api-access-jphl4\") pod \"test-curl-7257-139628158805312\" (UID: \"73f797d0-65af-45ae-81b2-59db31ac17a7\") " pod="opendatahub/test-curl-7257-139628158805312"
Jul 17 22:06:35.175132 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:35.175097    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jphl4\" (UniqueName: \"kubernetes.io/projected/73f797d0-65af-45ae-81b2-59db31ac17a7-kube-api-access-jphl4\") pod \"test-curl-7257-139628158805312\" (UID: \"73f797d0-65af-45ae-81b2-59db31ac17a7\") " pod="opendatahub/test-curl-7257-139628158805312"
Jul 17 22:06:35.182546 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:35.182520    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jphl4\" (UniqueName: \"kubernetes.io/projected/73f797d0-65af-45ae-81b2-59db31ac17a7-kube-api-access-jphl4\") pod \"test-curl-7257-139628158805312\" (UID: \"73f797d0-65af-45ae-81b2-59db31ac17a7\") " pod="opendatahub/test-curl-7257-139628158805312"
Jul 17 22:06:35.334666 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:35.334632    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158805312"
Jul 17 22:06:35.475948 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:35.475920    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-139628158805312"]
Jul 17 22:06:35.476076 ip-10-0-140-117 kubenswrapper[2569]: W0717 22:06:35.475946    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73f797d0_65af_45ae_81b2_59db31ac17a7.slice/crio-cfa35a116db881b90fab9fd5fcf54f0702445bfc2cf148239bc296e4ace8f830 WatchSource:0}: Error finding container cfa35a116db881b90fab9fd5fcf54f0702445bfc2cf148239bc296e4ace8f830: Status 404 returned error can't find the container with id cfa35a116db881b90fab9fd5fcf54f0702445bfc2cf148239bc296e4ace8f830
Jul 17 22:06:36.170974 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:36.170939    2569 generic.go:356] "Generic (PLEG): container finished" podID="73f797d0-65af-45ae-81b2-59db31ac17a7" containerID="1c454a6534180df68955254f73d6553ee285490c2277792eee4f0e8432ccf047" exitCode=0
Jul 17 22:06:36.171343 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:36.171029    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-139628158805312" event={"ID":"73f797d0-65af-45ae-81b2-59db31ac17a7","Type":"ContainerDied","Data":"1c454a6534180df68955254f73d6553ee285490c2277792eee4f0e8432ccf047"}
Jul 17 22:06:36.171343 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:36.171069    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-139628158805312" event={"ID":"73f797d0-65af-45ae-81b2-59db31ac17a7","Type":"ContainerStarted","Data":"cfa35a116db881b90fab9fd5fcf54f0702445bfc2cf148239bc296e4ace8f830"}
Jul 17 22:06:37.317526 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:37.317502    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158805312"
Jul 17 22:06:37.336691 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:37.336665    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7257-139628158805312_73f797d0-65af-45ae-81b2-59db31ac17a7/test-curl-7257-139628158805312/0.log"
Jul 17 22:06:37.368321 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:37.367561    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7257-139628158805312"]
Jul 17 22:06:37.373367 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:37.373340    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7257-139628158805312"]
Jul 17 22:06:37.492332 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:37.492234    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jphl4\" (UniqueName: \"kubernetes.io/projected/73f797d0-65af-45ae-81b2-59db31ac17a7-kube-api-access-jphl4\") pod \"73f797d0-65af-45ae-81b2-59db31ac17a7\" (UID: \"73f797d0-65af-45ae-81b2-59db31ac17a7\") "
Jul 17 22:06:37.494338 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:37.494299    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f797d0-65af-45ae-81b2-59db31ac17a7-kube-api-access-jphl4" (OuterVolumeSpecName: "kube-api-access-jphl4") pod "73f797d0-65af-45ae-81b2-59db31ac17a7" (UID: "73f797d0-65af-45ae-81b2-59db31ac17a7"). InnerVolumeSpecName "kube-api-access-jphl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:06:37.593236 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:37.593204    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jphl4\" (UniqueName: \"kubernetes.io/projected/73f797d0-65af-45ae-81b2-59db31ac17a7-kube-api-access-jphl4\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:06:38.183465 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:38.183437    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158805312"
Jul 17 22:06:38.183465 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:38.183447    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfa35a116db881b90fab9fd5fcf54f0702445bfc2cf148239bc296e4ace8f830"
Jul 17 22:06:38.192582 ip-10-0-140-117 kubenswrapper[2569]: E0717 22:06:38.192554    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7257-139628158805312\" is forbidden: User \"system:node:ip-10-0-140-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-117.ec2.internal' and this object" podUID="73f797d0-65af-45ae-81b2-59db31ac17a7" pod="opendatahub/test-curl-7257-139628158805312"
Jul 17 22:06:38.657113 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:38.657078    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73f797d0-65af-45ae-81b2-59db31ac17a7" path="/var/lib/kubelet/pods/73f797d0-65af-45ae-81b2-59db31ac17a7/volumes"
Jul 17 22:06:41.916242 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:41.916175    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc" podUID="0c896984-571c-42bf-819a-10e3cb6ee2f7" containerName="main" containerID="cri-o://b394f4b7b7fd35c2b6d73c724223d2b2e12f8d402aa68d8f95b2133036e58381" gracePeriod=45
Jul 17 22:06:42.180466 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.180388    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:06:42.208633 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.208603    2569 generic.go:356] "Generic (PLEG): container finished" podID="0c896984-571c-42bf-819a-10e3cb6ee2f7" containerID="b394f4b7b7fd35c2b6d73c724223d2b2e12f8d402aa68d8f95b2133036e58381" exitCode=0
Jul 17 22:06:42.208831 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.208643    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"
Jul 17 22:06:42.208831 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.208681    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc" event={"ID":"0c896984-571c-42bf-819a-10e3cb6ee2f7","Type":"ContainerDied","Data":"b394f4b7b7fd35c2b6d73c724223d2b2e12f8d402aa68d8f95b2133036e58381"}
Jul 17 22:06:42.208831 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.208723    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc" event={"ID":"0c896984-571c-42bf-819a-10e3cb6ee2f7","Type":"ContainerDied","Data":"5d663f224c16417f0482f9c1345d518aac1456a6c70cf4772ebc055a4df112bc"}
Jul 17 22:06:42.208831 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.208745    2569 scope.go:117] "RemoveContainer" containerID="b394f4b7b7fd35c2b6d73c724223d2b2e12f8d402aa68d8f95b2133036e58381"
Jul 17 22:06:42.218362 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.218344    2569 scope.go:117] "RemoveContainer" containerID="b394f4b7b7fd35c2b6d73c724223d2b2e12f8d402aa68d8f95b2133036e58381"
Jul 17 22:06:42.218648 ip-10-0-140-117 kubenswrapper[2569]: E0717 22:06:42.218630    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b394f4b7b7fd35c2b6d73c724223d2b2e12f8d402aa68d8f95b2133036e58381\": container with ID starting with b394f4b7b7fd35c2b6d73c724223d2b2e12f8d402aa68d8f95b2133036e58381 not found: ID does not exist" containerID="b394f4b7b7fd35c2b6d73c724223d2b2e12f8d402aa68d8f95b2133036e58381"
Jul 17 22:06:42.218742 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.218655    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b394f4b7b7fd35c2b6d73c724223d2b2e12f8d402aa68d8f95b2133036e58381"} err="failed to get container status \"b394f4b7b7fd35c2b6d73c724223d2b2e12f8d402aa68d8f95b2133036e58381\": rpc error: code = NotFound desc = could not find container \"b394f4b7b7fd35c2b6d73c724223d2b2e12f8d402aa68d8f95b2133036e58381\": container with ID starting with b394f4b7b7fd35c2b6d73c724223d2b2e12f8d402aa68d8f95b2133036e58381 not found: ID does not exist"
Jul 17 22:06:42.227166 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.227141    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swkv8\" (UniqueName: \"kubernetes.io/projected/0c896984-571c-42bf-819a-10e3cb6ee2f7-kube-api-access-swkv8\") pod \"0c896984-571c-42bf-819a-10e3cb6ee2f7\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") "
Jul 17 22:06:42.227295 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.227175    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-home\") pod \"0c896984-571c-42bf-819a-10e3cb6ee2f7\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") "
Jul 17 22:06:42.227295 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.227193    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-tmp-dir\") pod \"0c896984-571c-42bf-819a-10e3cb6ee2f7\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") "
Jul 17 22:06:42.227295 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.227264    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c896984-571c-42bf-819a-10e3cb6ee2f7-tls-certs\") pod \"0c896984-571c-42bf-819a-10e3cb6ee2f7\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") "
Jul 17 22:06:42.227295 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.227287    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-model-cache\") pod \"0c896984-571c-42bf-819a-10e3cb6ee2f7\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") "
Jul 17 22:06:42.227567 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.227312    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-dshm\") pod \"0c896984-571c-42bf-819a-10e3cb6ee2f7\" (UID: \"0c896984-571c-42bf-819a-10e3cb6ee2f7\") "
Jul 17 22:06:42.227567 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.227476    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c896984-571c-42bf-819a-10e3cb6ee2f7" (UID: "0c896984-571c-42bf-819a-10e3cb6ee2f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:06:42.227567 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.227486    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-home" (OuterVolumeSpecName: "home") pod "0c896984-571c-42bf-819a-10e3cb6ee2f7" (UID: "0c896984-571c-42bf-819a-10e3cb6ee2f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:06:42.227741 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.227620    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c896984-571c-42bf-819a-10e3cb6ee2f7" (UID: "0c896984-571c-42bf-819a-10e3cb6ee2f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:06:42.229476 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.229452    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-dshm" (OuterVolumeSpecName: "dshm") pod "0c896984-571c-42bf-819a-10e3cb6ee2f7" (UID: "0c896984-571c-42bf-819a-10e3cb6ee2f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:06:42.229756 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.229724    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c896984-571c-42bf-819a-10e3cb6ee2f7-kube-api-access-swkv8" (OuterVolumeSpecName: "kube-api-access-swkv8") pod "0c896984-571c-42bf-819a-10e3cb6ee2f7" (UID: "0c896984-571c-42bf-819a-10e3cb6ee2f7"). InnerVolumeSpecName "kube-api-access-swkv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:06:42.229855 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.229799    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c896984-571c-42bf-819a-10e3cb6ee2f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c896984-571c-42bf-819a-10e3cb6ee2f7" (UID: "0c896984-571c-42bf-819a-10e3cb6ee2f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 22:06:42.328844 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.328815    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swkv8\" (UniqueName: \"kubernetes.io/projected/0c896984-571c-42bf-819a-10e3cb6ee2f7-kube-api-access-swkv8\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:06:42.328844 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.328841    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-home\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:06:42.329019 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.328854    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-tmp-dir\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:06:42.329019 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.328869    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c896984-571c-42bf-819a-10e3cb6ee2f7-tls-certs\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:06:42.329019 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.328879    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-model-cache\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:06:42.329019 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.328888    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c896984-571c-42bf-819a-10e3cb6ee2f7-dshm\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:06:42.532834 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.532800    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"]
Jul 17 22:06:42.536005 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.535981    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ac333d/rate-test-model-ac333d-kserve-7c49d7fd95-rs2bc"]
Jul 17 22:06:42.657246 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.657214    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c896984-571c-42bf-819a-10e3cb6ee2f7" path="/var/lib/kubelet/pods/0c896984-571c-42bf-819a-10e3cb6ee2f7/volumes"
Jul 17 22:06:42.845032 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:42.844984    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p" podUID="57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d" containerName="main" containerID="cri-o://d7b68076fe86a0fe2ea5dc6ab32ab0ee78d7f0d61f36e9c5a549432d25714d81" gracePeriod=45
Jul 17 22:06:43.031960 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.031924    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7257-139628158806320"]
Jul 17 22:06:43.032559 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.032538    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73f797d0-65af-45ae-81b2-59db31ac17a7" containerName="test-curl-7257-139628158805312"
Jul 17 22:06:43.032668 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.032561    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="73f797d0-65af-45ae-81b2-59db31ac17a7" containerName="test-curl-7257-139628158805312"
Jul 17 22:06:43.032668 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.032606    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c896984-571c-42bf-819a-10e3cb6ee2f7" containerName="main"
Jul 17 22:06:43.032668 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.032616    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c896984-571c-42bf-819a-10e3cb6ee2f7" containerName="main"
Jul 17 22:06:43.032821 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.032684    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c896984-571c-42bf-819a-10e3cb6ee2f7" containerName="main"
Jul 17 22:06:43.032821 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.032693    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="73f797d0-65af-45ae-81b2-59db31ac17a7" containerName="test-curl-7257-139628158805312"
Jul 17 22:06:43.036054 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.036024    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158806320"
Jul 17 22:06:43.038135 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.038098    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rxnlj\""
Jul 17 22:06:43.041537 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.041512    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-139628158806320"]
Jul 17 22:06:43.103669 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.103601    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:06:43.136008 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.135978    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-home\") pod \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") "
Jul 17 22:06:43.136169 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.136026    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-model-cache\") pod \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") "
Jul 17 22:06:43.136169 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.136053    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h62wj\" (UniqueName: \"kubernetes.io/projected/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-kube-api-access-h62wj\") pod \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") "
Jul 17 22:06:43.136169 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.136075    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-tmp-dir\") pod \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") "
Jul 17 22:06:43.136169 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.136118    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-dshm\") pod \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") "
Jul 17 22:06:43.136169 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.136147    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-tls-certs\") pod \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\" (UID: \"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d\") "
Jul 17 22:06:43.136457 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.136314    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-model-cache" (OuterVolumeSpecName: "model-cache") pod "57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d" (UID: "57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:06:43.136457 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.136321    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x5jv\" (UniqueName: \"kubernetes.io/projected/79aa9d82-a163-4dc2-9636-3780b6eca82b-kube-api-access-9x5jv\") pod \"test-curl-7257-139628158806320\" (UID: \"79aa9d82-a163-4dc2-9636-3780b6eca82b\") " pod="opendatahub/test-curl-7257-139628158806320"
Jul 17 22:06:43.136457 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.136328    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-home" (OuterVolumeSpecName: "home") pod "57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d" (UID: "57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:06:43.136457 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.136365    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d" (UID: "57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:06:43.136662 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.136487    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-model-cache\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:06:43.136662 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.136504    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-tmp-dir\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:06:43.136662 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.136512    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-home\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:06:43.138336 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.138305    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-kube-api-access-h62wj" (OuterVolumeSpecName: "kube-api-access-h62wj") pod "57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d" (UID: "57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d"). InnerVolumeSpecName "kube-api-access-h62wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:06:43.138495 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.138339    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d" (UID: "57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 22:06:43.138495 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.138434    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-dshm" (OuterVolumeSpecName: "dshm") pod "57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d" (UID: "57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:06:43.217248 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.217211    2569 generic.go:356] "Generic (PLEG): container finished" podID="57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d" containerID="d7b68076fe86a0fe2ea5dc6ab32ab0ee78d7f0d61f36e9c5a549432d25714d81" exitCode=0
Jul 17 22:06:43.217426 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.217264    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"
Jul 17 22:06:43.217426 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.217269    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p" event={"ID":"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d","Type":"ContainerDied","Data":"d7b68076fe86a0fe2ea5dc6ab32ab0ee78d7f0d61f36e9c5a549432d25714d81"}
Jul 17 22:06:43.217426 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.217307    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p" event={"ID":"57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d","Type":"ContainerDied","Data":"fc67b3b757f059e81744412095219fdc75f61e1e6d293a2f94fd3ce0dfa7b908"}
Jul 17 22:06:43.217426 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.217330    2569 scope.go:117] "RemoveContainer" containerID="d7b68076fe86a0fe2ea5dc6ab32ab0ee78d7f0d61f36e9c5a549432d25714d81"
Jul 17 22:06:43.227155 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.227134    2569 scope.go:117] "RemoveContainer" containerID="d7b68076fe86a0fe2ea5dc6ab32ab0ee78d7f0d61f36e9c5a549432d25714d81"
Jul 17 22:06:43.227415 ip-10-0-140-117 kubenswrapper[2569]: E0717 22:06:43.227378    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b68076fe86a0fe2ea5dc6ab32ab0ee78d7f0d61f36e9c5a549432d25714d81\": container with ID starting with d7b68076fe86a0fe2ea5dc6ab32ab0ee78d7f0d61f36e9c5a549432d25714d81 not found: ID does not exist" containerID="d7b68076fe86a0fe2ea5dc6ab32ab0ee78d7f0d61f36e9c5a549432d25714d81"
Jul 17 22:06:43.227478 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.227421    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b68076fe86a0fe2ea5dc6ab32ab0ee78d7f0d61f36e9c5a549432d25714d81"} err="failed to get container status \"d7b68076fe86a0fe2ea5dc6ab32ab0ee78d7f0d61f36e9c5a549432d25714d81\": rpc error: code = NotFound desc = could not find container \"d7b68076fe86a0fe2ea5dc6ab32ab0ee78d7f0d61f36e9c5a549432d25714d81\": container with ID starting with d7b68076fe86a0fe2ea5dc6ab32ab0ee78d7f0d61f36e9c5a549432d25714d81 not found: ID does not exist"
Jul 17 22:06:43.237623 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.237598    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x5jv\" (UniqueName: \"kubernetes.io/projected/79aa9d82-a163-4dc2-9636-3780b6eca82b-kube-api-access-9x5jv\") pod \"test-curl-7257-139628158806320\" (UID: \"79aa9d82-a163-4dc2-9636-3780b6eca82b\") " pod="opendatahub/test-curl-7257-139628158806320"
Jul 17 22:06:43.237728 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.237644    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-dshm\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:06:43.237728 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.237656    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-tls-certs\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:06:43.237728 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.237665    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h62wj\" (UniqueName: \"kubernetes.io/projected/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d-kube-api-access-h62wj\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:06:43.238115 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.238095    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"]
Jul 17 22:06:43.242877 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.242844    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-051ea5/rate-test-model-051ea5-kserve-67c47dcd87-sf98p"]
Jul 17 22:06:43.245368 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.245351    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x5jv\" (UniqueName: \"kubernetes.io/projected/79aa9d82-a163-4dc2-9636-3780b6eca82b-kube-api-access-9x5jv\") pod \"test-curl-7257-139628158806320\" (UID: \"79aa9d82-a163-4dc2-9636-3780b6eca82b\") " pod="opendatahub/test-curl-7257-139628158806320"
Jul 17 22:06:43.349587 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.349556    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158806320"
Jul 17 22:06:43.484513 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:43.484488    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-139628158806320"]
Jul 17 22:06:43.486601 ip-10-0-140-117 kubenswrapper[2569]: W0717 22:06:43.486571    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79aa9d82_a163_4dc2_9636_3780b6eca82b.slice/crio-a98527e7305a880f32715edbd2da63f1d630e63f9624500447e252ce3a117911 WatchSource:0}: Error finding container a98527e7305a880f32715edbd2da63f1d630e63f9624500447e252ce3a117911: Status 404 returned error can't find the container with id a98527e7305a880f32715edbd2da63f1d630e63f9624500447e252ce3a117911
Jul 17 22:06:44.227285 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:44.227251    2569 generic.go:356] "Generic (PLEG): container finished" podID="79aa9d82-a163-4dc2-9636-3780b6eca82b" containerID="ed41c68e3461186647b3af021376d8a4142c4312668b26cd3cb032d2a8bb2c6c" exitCode=0
Jul 17 22:06:44.227675 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:44.227335    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-139628158806320" event={"ID":"79aa9d82-a163-4dc2-9636-3780b6eca82b","Type":"ContainerDied","Data":"ed41c68e3461186647b3af021376d8a4142c4312668b26cd3cb032d2a8bb2c6c"}
Jul 17 22:06:44.227675 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:44.227374    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-139628158806320" event={"ID":"79aa9d82-a163-4dc2-9636-3780b6eca82b","Type":"ContainerStarted","Data":"a98527e7305a880f32715edbd2da63f1d630e63f9624500447e252ce3a117911"}
Jul 17 22:06:44.657897 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:44.657864    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d" path="/var/lib/kubelet/pods/57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d/volumes"
Jul 17 22:06:45.377972 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:45.377946    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158806320"
Jul 17 22:06:45.397170 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:45.397144    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7257-139628158806320_79aa9d82-a163-4dc2-9636-3780b6eca82b/test-curl-7257-139628158806320/0.log"
Jul 17 22:06:45.419034 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:45.418999    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7257-139628158806320"]
Jul 17 22:06:45.422503 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:45.422476    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7257-139628158806320"]
Jul 17 22:06:45.459176 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:45.459145    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x5jv\" (UniqueName: \"kubernetes.io/projected/79aa9d82-a163-4dc2-9636-3780b6eca82b-kube-api-access-9x5jv\") pod \"79aa9d82-a163-4dc2-9636-3780b6eca82b\" (UID: \"79aa9d82-a163-4dc2-9636-3780b6eca82b\") "
Jul 17 22:06:45.461294 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:45.461265    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79aa9d82-a163-4dc2-9636-3780b6eca82b-kube-api-access-9x5jv" (OuterVolumeSpecName: "kube-api-access-9x5jv") pod "79aa9d82-a163-4dc2-9636-3780b6eca82b" (UID: "79aa9d82-a163-4dc2-9636-3780b6eca82b"). InnerVolumeSpecName "kube-api-access-9x5jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:06:45.560011 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:45.559922    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x5jv\" (UniqueName: \"kubernetes.io/projected/79aa9d82-a163-4dc2-9636-3780b6eca82b-kube-api-access-9x5jv\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:06:46.141714 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:46.141603    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:06:46.240681 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:46.240648    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a98527e7305a880f32715edbd2da63f1d630e63f9624500447e252ce3a117911"
Jul 17 22:06:46.240917 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:46.240690    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-139628158806320"
Jul 17 22:06:46.241511 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:46.241470    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:06:46.250713 ip-10-0-140-117 kubenswrapper[2569]: E0717 22:06:46.250689    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7257-139628158806320\" is forbidden: User \"system:node:ip-10-0-140-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-117.ec2.internal' and this object" podUID="79aa9d82-a163-4dc2-9636-3780b6eca82b" pod="opendatahub/test-curl-7257-139628158806320"
Jul 17 22:06:46.654173 ip-10-0-140-117 kubenswrapper[2569]: E0717 22:06:46.654136    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7257-139628158806320\" is forbidden: User \"system:node:ip-10-0-140-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-117.ec2.internal' and this object" podUID="79aa9d82-a163-4dc2-9636-3780b6eca82b" pod="opendatahub/test-curl-7257-139628158806320"
Jul 17 22:06:46.655954 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:46.655925    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79aa9d82-a163-4dc2-9636-3780b6eca82b" path="/var/lib/kubelet/pods/79aa9d82-a163-4dc2-9636-3780b6eca82b/volumes"
Jul 17 22:06:56.206206 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:56.206171    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:06:59.710526 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:06:59.710493    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 22:07:00.705220 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:00.705173    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:07:06.667055 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:06.667019    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:07:06.738735 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:06.738701    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:07:11.374628 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:11.373124    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:07:11.589511 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:11.589480    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:07:11.700984 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:11.700901    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-051ea5-v8nmc"]
Jul 17 22:07:11.701366 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:11.701347    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79aa9d82-a163-4dc2-9636-3780b6eca82b" containerName="test-curl-7257-139628158806320"
Jul 17 22:07:11.701497 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:11.701369    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="79aa9d82-a163-4dc2-9636-3780b6eca82b" containerName="test-curl-7257-139628158806320"
Jul 17 22:07:11.701497 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:11.701460    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d" containerName="main"
Jul 17 22:07:11.701497 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:11.701471    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d" containerName="main"
Jul 17 22:07:11.701652 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:11.701592    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="57cbb9fb-ea20-44e9-af6f-dd59f4d7fa9d" containerName="main"
Jul 17 22:07:11.701652 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:11.701603    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="79aa9d82-a163-4dc2-9636-3780b6eca82b" containerName="test-curl-7257-139628158806320"
Jul 17 22:07:11.706825 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:11.706803    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-051ea5-v8nmc"
Jul 17 22:07:11.709080 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:11.709060    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 22:07:11.711908 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:11.711883    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-051ea5-v8nmc"]
Jul 17 22:07:11.801196 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:11.801158    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07cca88b-5603-4960-bea9-079bfdafdd48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-051ea5-v8nmc\" (UID: \"07cca88b-5603-4960-bea9-079bfdafdd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-051ea5-v8nmc"
Jul 17 22:07:11.902818 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:11.902772    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07cca88b-5603-4960-bea9-079bfdafdd48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-051ea5-v8nmc\" (UID: \"07cca88b-5603-4960-bea9-079bfdafdd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-051ea5-v8nmc"
Jul 17 22:07:11.903438 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:11.903387    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07cca88b-5603-4960-bea9-079bfdafdd48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-051ea5-v8nmc\" (UID: \"07cca88b-5603-4960-bea9-079bfdafdd48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-051ea5-v8nmc"
Jul 17 22:07:12.019394 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:12.019312    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-051ea5-v8nmc"
Jul 17 22:07:12.168761 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:12.168730    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-051ea5-v8nmc"]
Jul 17 22:07:12.170244 ip-10-0-140-117 kubenswrapper[2569]: W0717 22:07:12.170202    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07cca88b_5603_4960_bea9_079bfdafdd48.slice/crio-b1f8b7b50cca7f3480dacd9e61444152cf8268d20c8b12f8dc1dd26cc7956240 WatchSource:0}: Error finding container b1f8b7b50cca7f3480dacd9e61444152cf8268d20c8b12f8dc1dd26cc7956240: Status 404 returned error can't find the container with id b1f8b7b50cca7f3480dacd9e61444152cf8268d20c8b12f8dc1dd26cc7956240
Jul 17 22:07:12.223039 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:12.223016    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:07:12.323138 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:12.323056    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:07:12.398914 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:12.398883    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-051ea5-v8nmc" event={"ID":"07cca88b-5603-4960-bea9-079bfdafdd48","Type":"ContainerStarted","Data":"b1f8b7b50cca7f3480dacd9e61444152cf8268d20c8b12f8dc1dd26cc7956240"}
Jul 17 22:07:13.406393 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:13.406360    2569 generic.go:356] "Generic (PLEG): container finished" podID="07cca88b-5603-4960-bea9-079bfdafdd48" containerID="78ba8a372ed8587c736553a129171cae598135572370489a9d658b663a8d0f88" exitCode=0
Jul 17 22:07:13.406804 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:13.406449    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-051ea5-v8nmc" event={"ID":"07cca88b-5603-4960-bea9-079bfdafdd48","Type":"ContainerDied","Data":"78ba8a372ed8587c736553a129171cae598135572370489a9d658b663a8d0f88"}
Jul 17 22:07:14.553850 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:14.553826    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-051ea5-v8nmc"
Jul 17 22:07:14.726043 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:14.725966    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07cca88b-5603-4960-bea9-079bfdafdd48-maas-api-service-ca\") pod \"07cca88b-5603-4960-bea9-079bfdafdd48\" (UID: \"07cca88b-5603-4960-bea9-079bfdafdd48\") "
Jul 17 22:07:14.726349 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:14.726328    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07cca88b-5603-4960-bea9-079bfdafdd48-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "07cca88b-5603-4960-bea9-079bfdafdd48" (UID: "07cca88b-5603-4960-bea9-079bfdafdd48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:07:14.826843 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:14.826815    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07cca88b-5603-4960-bea9-079bfdafdd48-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:07:15.418807 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:15.418774    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-051ea5-v8nmc"
Jul 17 22:07:15.418995 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:15.418780    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-051ea5-v8nmc" event={"ID":"07cca88b-5603-4960-bea9-079bfdafdd48","Type":"ContainerDied","Data":"b1f8b7b50cca7f3480dacd9e61444152cf8268d20c8b12f8dc1dd26cc7956240"}
Jul 17 22:07:15.418995 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:15.418910    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1f8b7b50cca7f3480dacd9e61444152cf8268d20c8b12f8dc1dd26cc7956240"
Jul 17 22:07:22.229762 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:22.229724    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"]
Jul 17 22:07:22.230235 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:22.229970    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9" podUID="4788f201-3d97-4c2e-ba51-7a38df63b411" containerName="maas-api" containerID="cri-o://9e706c3abe6f2d2c9c7fd22c45385146da0e366091fca1673a36f0d273df4af9" gracePeriod=30
Jul 17 22:07:22.260713 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:22.260661    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:07:22.692813 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:22.692786    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"
Jul 17 22:07:22.796685 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:22.796587    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67rpc\" (UniqueName: \"kubernetes.io/projected/4788f201-3d97-4c2e-ba51-7a38df63b411-kube-api-access-67rpc\") pod \"4788f201-3d97-4c2e-ba51-7a38df63b411\" (UID: \"4788f201-3d97-4c2e-ba51-7a38df63b411\") "
Jul 17 22:07:22.796685 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:22.796631    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4788f201-3d97-4c2e-ba51-7a38df63b411-openshift-service-ca\") pod \"4788f201-3d97-4c2e-ba51-7a38df63b411\" (UID: \"4788f201-3d97-4c2e-ba51-7a38df63b411\") "
Jul 17 22:07:22.796927 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:22.796776    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4788f201-3d97-4c2e-ba51-7a38df63b411-maas-api-tls\") pod \"4788f201-3d97-4c2e-ba51-7a38df63b411\" (UID: \"4788f201-3d97-4c2e-ba51-7a38df63b411\") "
Jul 17 22:07:22.797105 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:22.797074    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4788f201-3d97-4c2e-ba51-7a38df63b411-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4788f201-3d97-4c2e-ba51-7a38df63b411" (UID: "4788f201-3d97-4c2e-ba51-7a38df63b411"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:07:22.798952 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:22.798915    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4788f201-3d97-4c2e-ba51-7a38df63b411-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4788f201-3d97-4c2e-ba51-7a38df63b411" (UID: "4788f201-3d97-4c2e-ba51-7a38df63b411"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 22:07:22.798952 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:22.798929    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4788f201-3d97-4c2e-ba51-7a38df63b411-kube-api-access-67rpc" (OuterVolumeSpecName: "kube-api-access-67rpc") pod "4788f201-3d97-4c2e-ba51-7a38df63b411" (UID: "4788f201-3d97-4c2e-ba51-7a38df63b411"). InnerVolumeSpecName "kube-api-access-67rpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:07:22.897530 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:22.897485    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4788f201-3d97-4c2e-ba51-7a38df63b411-maas-api-tls\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:07:22.897530 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:22.897524    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67rpc\" (UniqueName: \"kubernetes.io/projected/4788f201-3d97-4c2e-ba51-7a38df63b411-kube-api-access-67rpc\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:07:22.897743 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:22.897541    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4788f201-3d97-4c2e-ba51-7a38df63b411-openshift-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:07:23.467952 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:23.467917    2569 generic.go:356] "Generic (PLEG): container finished" podID="4788f201-3d97-4c2e-ba51-7a38df63b411" containerID="9e706c3abe6f2d2c9c7fd22c45385146da0e366091fca1673a36f0d273df4af9" exitCode=0
Jul 17 22:07:23.468449 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:23.467970    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"
Jul 17 22:07:23.468449 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:23.467999    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9" event={"ID":"4788f201-3d97-4c2e-ba51-7a38df63b411","Type":"ContainerDied","Data":"9e706c3abe6f2d2c9c7fd22c45385146da0e366091fca1673a36f0d273df4af9"}
Jul 17 22:07:23.468449 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:23.468051    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9" event={"ID":"4788f201-3d97-4c2e-ba51-7a38df63b411","Type":"ContainerDied","Data":"eba49c838cdbee325434ae41b334b396687461ad8f8921b9ecd61b5cba7f836e"}
Jul 17 22:07:23.468449 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:23.468073    2569 scope.go:117] "RemoveContainer" containerID="9e706c3abe6f2d2c9c7fd22c45385146da0e366091fca1673a36f0d273df4af9"
Jul 17 22:07:23.478502 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:23.478479    2569 scope.go:117] "RemoveContainer" containerID="9e706c3abe6f2d2c9c7fd22c45385146da0e366091fca1673a36f0d273df4af9"
Jul 17 22:07:23.478767 ip-10-0-140-117 kubenswrapper[2569]: E0717 22:07:23.478746    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e706c3abe6f2d2c9c7fd22c45385146da0e366091fca1673a36f0d273df4af9\": container with ID starting with 9e706c3abe6f2d2c9c7fd22c45385146da0e366091fca1673a36f0d273df4af9 not found: ID does not exist" containerID="9e706c3abe6f2d2c9c7fd22c45385146da0e366091fca1673a36f0d273df4af9"
Jul 17 22:07:23.478849 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:23.478779    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e706c3abe6f2d2c9c7fd22c45385146da0e366091fca1673a36f0d273df4af9"} err="failed to get container status \"9e706c3abe6f2d2c9c7fd22c45385146da0e366091fca1673a36f0d273df4af9\": rpc error: code = NotFound desc = could not find container \"9e706c3abe6f2d2c9c7fd22c45385146da0e366091fca1673a36f0d273df4af9\": container with ID starting with 9e706c3abe6f2d2c9c7fd22c45385146da0e366091fca1673a36f0d273df4af9 not found: ID does not exist"
Jul 17 22:07:23.489553 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:23.489520    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"]
Jul 17 22:07:23.493069 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:23.493047    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-051ea5-74ccc4895-sftd9"]
Jul 17 22:07:24.656446 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:24.656384    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4788f201-3d97-4c2e-ba51-7a38df63b411" path="/var/lib/kubelet/pods/4788f201-3d97-4c2e-ba51-7a38df63b411/volumes"
Jul 17 22:07:25.212537 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:25.212498    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jb626"]
Jul 17 22:07:26.784177 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:26.784142    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:07:32.670287 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:32.670247    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:07:32.771039 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:32.771006    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:07:37.492561 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:37.492514    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:07:37.690463 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:37.690424    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:07:38.124101 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:38.124070    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84b6ff4847-rd2kz_e390f181-7b81-4dc9-96f4-a94056ecdbdc/authorino/0.log"
Jul 17 22:07:40.203372 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:40.203337    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj"]
Jul 17 22:07:40.203806 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:40.203756    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07cca88b-5603-4960-bea9-079bfdafdd48" containerName="revoke-keys"
Jul 17 22:07:40.203806 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:40.203771    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="07cca88b-5603-4960-bea9-079bfdafdd48" containerName="revoke-keys"
Jul 17 22:07:40.203806 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:40.203785    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4788f201-3d97-4c2e-ba51-7a38df63b411" containerName="maas-api"
Jul 17 22:07:40.203806 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:40.203794    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4788f201-3d97-4c2e-ba51-7a38df63b411" containerName="maas-api"
Jul 17 22:07:40.203952 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:40.203894    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4788f201-3d97-4c2e-ba51-7a38df63b411" containerName="maas-api"
Jul 17 22:07:40.203952 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:40.203902    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="07cca88b-5603-4960-bea9-079bfdafdd48" containerName="revoke-keys"
Jul 17 22:07:40.208257 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:40.208239    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj"
Jul 17 22:07:40.210822 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:40.210797    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:07:40.210955 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:40.210797    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 22:07:40.213850 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:40.213823    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj"]
Jul 17 22:07:40.238574 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:40.238543    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85db2a77-e6b6-4305-be81-3af9c2518ff2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj\" (UID: \"85db2a77-e6b6-4305-be81-3af9c2518ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj"
Jul 17 22:07:40.339602 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:40.339553    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85db2a77-e6b6-4305-be81-3af9c2518ff2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj\" (UID: \"85db2a77-e6b6-4305-be81-3af9c2518ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj"
Jul 17 22:07:40.340207 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:40.340186    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85db2a77-e6b6-4305-be81-3af9c2518ff2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj\" (UID: \"85db2a77-e6b6-4305-be81-3af9c2518ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj"
Jul 17 22:07:40.520861 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:40.520780    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj"
Jul 17 22:07:40.657267 ip-10-0-140-117 kubenswrapper[2569]: W0717 22:07:40.657238    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85db2a77_e6b6_4305_be81_3af9c2518ff2.slice/crio-c8176633e84f751fed5775cabf7ce78c967ceaa4f1735cafb0707d113da65790 WatchSource:0}: Error finding container c8176633e84f751fed5775cabf7ce78c967ceaa4f1735cafb0707d113da65790: Status 404 returned error can't find the container with id c8176633e84f751fed5775cabf7ce78c967ceaa4f1735cafb0707d113da65790
Jul 17 22:07:40.658148 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:40.658126    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj"]
Jul 17 22:07:41.591379 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:41.591345    2569 generic.go:356] "Generic (PLEG): container finished" podID="85db2a77-e6b6-4305-be81-3af9c2518ff2" containerID="02803e7e7e7081b9196365cde06a28befb2142fa49ae865afe5a83c82798b1e4" exitCode=6
Jul 17 22:07:41.591792 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:41.591433    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj" event={"ID":"85db2a77-e6b6-4305-be81-3af9c2518ff2","Type":"ContainerDied","Data":"02803e7e7e7081b9196365cde06a28befb2142fa49ae865afe5a83c82798b1e4"}
Jul 17 22:07:41.591792 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:41.591523    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj" event={"ID":"85db2a77-e6b6-4305-be81-3af9c2518ff2","Type":"ContainerStarted","Data":"c8176633e84f751fed5775cabf7ce78c967ceaa4f1735cafb0707d113da65790"}
Jul 17 22:07:41.591792 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:41.591631    2569 scope.go:117] "RemoveContainer" containerID="02803e7e7e7081b9196365cde06a28befb2142fa49ae865afe5a83c82798b1e4"
Jul 17 22:07:42.598909 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:42.598873    2569 generic.go:356] "Generic (PLEG): container finished" podID="85db2a77-e6b6-4305-be81-3af9c2518ff2" containerID="fa7fade868bebcb5d2d8eeacaee831f887695fe2876c953aaee6cb6d9bd9e762" exitCode=6
Jul 17 22:07:42.599364 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:42.598955    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj" event={"ID":"85db2a77-e6b6-4305-be81-3af9c2518ff2","Type":"ContainerDied","Data":"fa7fade868bebcb5d2d8eeacaee831f887695fe2876c953aaee6cb6d9bd9e762"}
Jul 17 22:07:42.599364 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:42.599023    2569 scope.go:117] "RemoveContainer" containerID="02803e7e7e7081b9196365cde06a28befb2142fa49ae865afe5a83c82798b1e4"
Jul 17 22:07:42.599364 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:42.599117    2569 scope.go:117] "RemoveContainer" containerID="fa7fade868bebcb5d2d8eeacaee831f887695fe2876c953aaee6cb6d9bd9e762"
Jul 17 22:07:42.599364 ip-10-0-140-117 kubenswrapper[2569]: E0717 22:07:42.599344    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj_odh-ai-gateway-infra(85db2a77-e6b6-4305-be81-3af9c2518ff2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj" podUID="85db2a77-e6b6-4305-be81-3af9c2518ff2"
Jul 17 22:07:43.216855 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:43.216826    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-swflf_7134f6e3-e6cc-48f0-85c8-87dd703137d8/manager/0.log"
Jul 17 22:07:43.335695 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:43.335651    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-l4wzj_e4c502b9-5cc5-40d3-ac93-d52a0f629279/manager/0.log"
Jul 17 22:07:43.560017 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:43.559937    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-lms8h_2106c72f-6b22-4c1b-8328-9fb93b711349/server/0.log"
Jul 17 22:07:43.607710 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:43.607668    2569 scope.go:117] "RemoveContainer" containerID="fa7fade868bebcb5d2d8eeacaee831f887695fe2876c953aaee6cb6d9bd9e762"
Jul 17 22:07:43.608074 ip-10-0-140-117 kubenswrapper[2569]: E0717 22:07:43.607886    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj_odh-ai-gateway-infra(85db2a77-e6b6-4305-be81-3af9c2518ff2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj" podUID="85db2a77-e6b6-4305-be81-3af9c2518ff2"
Jul 17 22:07:43.680777 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:43.680745    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-r8sf5_aaf6b929-c2d5-48fb-ab6b-535386eb9635/manager/0.log"
Jul 17 22:07:43.801900 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:43.801872    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b8f6b79df-jqdz6_abe9dd49-3a15-4f58-b19f-e871479de840/copy-manifests/0.log"
Jul 17 22:07:43.830529 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:43.830507    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b8f6b79df-jqdz6_abe9dd49-3a15-4f58-b19f-e871479de840/manager/0.log"
Jul 17 22:07:44.532678 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:44.532647    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738730-qqw5r_817f33cb-969d-467c-8b7b-36215e895153/cleanup/0.log"
Jul 17 22:07:44.638170 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:44.638146    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738745-qfv5x_a6b3e52b-5945-4c17-8bb7-92aa7bb119e6/cleanup/0.log"
Jul 17 22:07:45.167526 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:45.167494    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-786043-gcxdq_38ef5926-5fd2-4128-8c58-76dba88ccc77/revoke-keys/0.log"
Jul 17 22:07:45.277068 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:45.277038    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj_85db2a77-e6b6-4305-be81-3af9c2518ff2/revoke-keys/1.log"
Jul 17 22:07:45.386971 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:45.386941    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-62f6c7d6-mxzcp_d7d39f53-5de5-4702-baaa-a02b744153c1/revoke-keys/0.log"
Jul 17 22:07:45.496884 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:45.496791    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-c39f8651-m2972_e25088f2-a9c7-414c-951a-919f158b5565/revoke-keys/0.log"
Jul 17 22:07:45.709129 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:45.709098    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-051ea5-v8nmc_07cca88b-5603-4960-bea9-079bfdafdd48/revoke-keys/0.log"
Jul 17 22:07:46.022549 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.022513    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb"]
Jul 17 22:07:46.027382 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.027365    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb"
Jul 17 22:07:46.035246 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.035218    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb"]
Jul 17 22:07:46.087694 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.087645    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65d4c52f-6bb8-46d3-81a9-c834668f2d66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb\" (UID: \"65d4c52f-6bb8-46d3-81a9-c834668f2d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb"
Jul 17 22:07:46.188297 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.188263    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65d4c52f-6bb8-46d3-81a9-c834668f2d66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb\" (UID: \"65d4c52f-6bb8-46d3-81a9-c834668f2d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb"
Jul 17 22:07:46.188955 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.188936    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65d4c52f-6bb8-46d3-81a9-c834668f2d66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb\" (UID: \"65d4c52f-6bb8-46d3-81a9-c834668f2d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb"
Jul 17 22:07:46.340179 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.340143    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb"
Jul 17 22:07:46.480260 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.480229    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb"]
Jul 17 22:07:46.481598 ip-10-0-140-117 kubenswrapper[2569]: W0717 22:07:46.481575    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d4c52f_6bb8_46d3_81a9_c834668f2d66.slice/crio-0dea0f980306037b43d30186c5744b6e32736095fc6bf5994aeb49e958bba3dd WatchSource:0}: Error finding container 0dea0f980306037b43d30186c5744b6e32736095fc6bf5994aeb49e958bba3dd: Status 404 returned error can't find the container with id 0dea0f980306037b43d30186c5744b6e32736095fc6bf5994aeb49e958bba3dd
Jul 17 22:07:46.556615 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.556577    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2_0bd5eb95-6b65-4942-b629-5939d4c8e214/util/0.log"
Jul 17 22:07:46.562804 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.562774    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2_0bd5eb95-6b65-4942-b629-5939d4c8e214/pull/0.log"
Jul 17 22:07:46.568228 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.568201    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2_0bd5eb95-6b65-4942-b629-5939d4c8e214/extract/0.log"
Jul 17 22:07:46.626505 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.626420    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb" event={"ID":"65d4c52f-6bb8-46d3-81a9-c834668f2d66","Type":"ContainerStarted","Data":"0dea0f980306037b43d30186c5744b6e32736095fc6bf5994aeb49e958bba3dd"}
Jul 17 22:07:46.675058 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.675031    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb_c82aca7c-ff06-4c79-8c14-198701662c0d/util/0.log"
Jul 17 22:07:46.679825 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.679799    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb_c82aca7c-ff06-4c79-8c14-198701662c0d/pull/0.log"
Jul 17 22:07:46.684354 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.684332    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb_c82aca7c-ff06-4c79-8c14-198701662c0d/extract/0.log"
Jul 17 22:07:46.795741 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.795684    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b_82253133-dd19-4596-ac01-b7dd1e343483/pull/0.log"
Jul 17 22:07:46.800969 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.800949    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b_82253133-dd19-4596-ac01-b7dd1e343483/extract/0.log"
Jul 17 22:07:46.805621 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.805603    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b_82253133-dd19-4596-ac01-b7dd1e343483/util/0.log"
Jul 17 22:07:46.904776 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.904694    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d_3d21dbc9-6449-43d8-a58b-40e1333053ec/util/0.log"
Jul 17 22:07:46.909546 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.909512    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d_3d21dbc9-6449-43d8-a58b-40e1333053ec/pull/0.log"
Jul 17 22:07:46.915974 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:46.915946    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d_3d21dbc9-6449-43d8-a58b-40e1333053ec/extract/0.log"
Jul 17 22:07:47.042927 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:47.042886    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84b6ff4847-rd2kz_e390f181-7b81-4dc9-96f4-a94056ecdbdc/authorino/0.log"
Jul 17 22:07:47.157532 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:47.157459    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-2m5tx_07cffc65-fcdd-49b3-922f-c5af96a7070e/manager/0.log"
Jul 17 22:07:47.267027 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:47.266994    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-b9bnv_a7a6ffaf-914c-475b-9acc-f07241caca1d/manager/0.log"
Jul 17 22:07:47.374839 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:47.374808    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-dljpp_a09d35c4-d15f-45ad-855f-1b43f8fac1bf/kuadrant-console-plugin/0.log"
Jul 17 22:07:47.499020 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:47.498942    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-rtbw9_b856ffec-b5f2-4f2a-b3c8-80f28abb5860/registry-server/0.log"
Jul 17 22:07:47.623058 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:47.623020    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-n42th_e5248760-056f-4372-8617-4b4c22687016/manager/0.log"
Jul 17 22:07:47.633508 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:47.633480    2569 generic.go:356] "Generic (PLEG): container finished" podID="65d4c52f-6bb8-46d3-81a9-c834668f2d66" containerID="0e4a5a08afe021d1eb6f74e876a0d8e08c09a2c0d8ada789756a5f869016df6f" exitCode=6
Jul 17 22:07:47.633680 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:47.633577    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb" event={"ID":"65d4c52f-6bb8-46d3-81a9-c834668f2d66","Type":"ContainerDied","Data":"0e4a5a08afe021d1eb6f74e876a0d8e08c09a2c0d8ada789756a5f869016df6f"}
Jul 17 22:07:47.633784 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:47.633768    2569 scope.go:117] "RemoveContainer" containerID="0e4a5a08afe021d1eb6f74e876a0d8e08c09a2c0d8ada789756a5f869016df6f"
Jul 17 22:07:47.733128 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:47.733104    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jb626_6b6bc014-05c4-4303-88c4-8536484d2681/limitador/0.log"
Jul 17 22:07:47.849373 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:47.849346    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-4gb9m_e4124a45-e1e0-4c94-b4f0-bf65a28e54bd/manager/0.log"
Jul 17 22:07:48.391308 ip-10-0-140-117 kubenswrapper[2569]: E0717 22:07:48.391273    2569 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d4c52f_6bb8_46d3_81a9_c834668f2d66.slice/crio-956fd7cbb93e27c3f646034f71b3293e227bcf7d7348e1b0efac9b1cc1fe4126.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d4c52f_6bb8_46d3_81a9_c834668f2d66.slice/crio-conmon-956fd7cbb93e27c3f646034f71b3293e227bcf7d7348e1b0efac9b1cc1fe4126.scope\": RecentStats: unable to find data in memory cache]"
Jul 17 22:07:48.541170 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:48.541142    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cf476465c-wbzp5_efda4d41-9563-4e36-b789-859c4786e297/kube-auth-proxy/0.log"
Jul 17 22:07:48.641043 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:48.640959    2569 generic.go:356] "Generic (PLEG): container finished" podID="65d4c52f-6bb8-46d3-81a9-c834668f2d66" containerID="956fd7cbb93e27c3f646034f71b3293e227bcf7d7348e1b0efac9b1cc1fe4126" exitCode=6
Jul 17 22:07:48.641198 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:48.641068    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb" event={"ID":"65d4c52f-6bb8-46d3-81a9-c834668f2d66","Type":"ContainerDied","Data":"956fd7cbb93e27c3f646034f71b3293e227bcf7d7348e1b0efac9b1cc1fe4126"}
Jul 17 22:07:48.641198 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:48.641125    2569 scope.go:117] "RemoveContainer" containerID="0e4a5a08afe021d1eb6f74e876a0d8e08c09a2c0d8ada789756a5f869016df6f"
Jul 17 22:07:48.641325 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:48.641239    2569 scope.go:117] "RemoveContainer" containerID="956fd7cbb93e27c3f646034f71b3293e227bcf7d7348e1b0efac9b1cc1fe4126"
Jul 17 22:07:48.641465 ip-10-0-140-117 kubenswrapper[2569]: E0717 22:07:48.641445    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb_odh-ai-gateway-infra(65d4c52f-6bb8-46d3-81a9-c834668f2d66)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb" podUID="65d4c52f-6bb8-46d3-81a9-c834668f2d66"
Jul 17 22:07:48.984694 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:48.984615    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-756f96dd8-2m7s2_2cfce8d6-1da4-4003-8183-31744592793b/router/0.log"
Jul 17 22:07:49.648689 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:49.648663    2569 scope.go:117] "RemoveContainer" containerID="956fd7cbb93e27c3f646034f71b3293e227bcf7d7348e1b0efac9b1cc1fe4126"
Jul 17 22:07:49.648868 ip-10-0-140-117 kubenswrapper[2569]: E0717 22:07:49.648853    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb_odh-ai-gateway-infra(65d4c52f-6bb8-46d3-81a9-c834668f2d66)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb" podUID="65d4c52f-6bb8-46d3-81a9-c834668f2d66"
Jul 17 22:07:49.880094 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:49.880063    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-jlgjd_114c61c3-3373-4ae2-b9cd-7acc63966e67/main/0.log"
Jul 17 22:07:54.708743 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:54.708706    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4qhz6/must-gather-4lwwb"]
Jul 17 22:07:54.712453 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:54.712436    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4qhz6/must-gather-4lwwb"
Jul 17 22:07:54.714572 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:54.714544    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4qhz6\"/\"kube-root-ca.crt\""
Jul 17 22:07:54.714710 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:54.714650    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4qhz6\"/\"openshift-service-ca.crt\""
Jul 17 22:07:54.719963 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:54.719940    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4qhz6/must-gather-4lwwb"]
Jul 17 22:07:54.730685 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:54.730649    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:07:54.760887 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:54.760862    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f2d55ae4-0b02-4c66-a41d-8bb14dcb7307-must-gather-output\") pod \"must-gather-4lwwb\" (UID: \"f2d55ae4-0b02-4c66-a41d-8bb14dcb7307\") " pod="openshift-must-gather-4qhz6/must-gather-4lwwb"
Jul 17 22:07:54.761038 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:54.760908    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m87mv\" (UniqueName: \"kubernetes.io/projected/f2d55ae4-0b02-4c66-a41d-8bb14dcb7307-kube-api-access-m87mv\") pod \"must-gather-4lwwb\" (UID: \"f2d55ae4-0b02-4c66-a41d-8bb14dcb7307\") " pod="openshift-must-gather-4qhz6/must-gather-4lwwb"
Jul 17 22:07:54.862288 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:54.862252    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f2d55ae4-0b02-4c66-a41d-8bb14dcb7307-must-gather-output\") pod \"must-gather-4lwwb\" (UID: \"f2d55ae4-0b02-4c66-a41d-8bb14dcb7307\") " pod="openshift-must-gather-4qhz6/must-gather-4lwwb"
Jul 17 22:07:54.862484 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:54.862313    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m87mv\" (UniqueName: \"kubernetes.io/projected/f2d55ae4-0b02-4c66-a41d-8bb14dcb7307-kube-api-access-m87mv\") pod \"must-gather-4lwwb\" (UID: \"f2d55ae4-0b02-4c66-a41d-8bb14dcb7307\") " pod="openshift-must-gather-4qhz6/must-gather-4lwwb"
Jul 17 22:07:54.862668 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:54.862646    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f2d55ae4-0b02-4c66-a41d-8bb14dcb7307-must-gather-output\") pod \"must-gather-4lwwb\" (UID: \"f2d55ae4-0b02-4c66-a41d-8bb14dcb7307\") " pod="openshift-must-gather-4qhz6/must-gather-4lwwb"
Jul 17 22:07:54.869904 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:54.869883    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m87mv\" (UniqueName: \"kubernetes.io/projected/f2d55ae4-0b02-4c66-a41d-8bb14dcb7307-kube-api-access-m87mv\") pod \"must-gather-4lwwb\" (UID: \"f2d55ae4-0b02-4c66-a41d-8bb14dcb7307\") " pod="openshift-must-gather-4qhz6/must-gather-4lwwb"
Jul 17 22:07:55.022103 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:55.022019    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4qhz6/must-gather-4lwwb"
Jul 17 22:07:55.163119 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:55.163091    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4qhz6/must-gather-4lwwb"]
Jul 17 22:07:55.164499 ip-10-0-140-117 kubenswrapper[2569]: W0717 22:07:55.164474    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2d55ae4_0b02_4c66_a41d_8bb14dcb7307.slice/crio-d505fa00bd30795b6b8d4dcaff6231432149b2f8b863677ef95af1d9843d27f6 WatchSource:0}: Error finding container d505fa00bd30795b6b8d4dcaff6231432149b2f8b863677ef95af1d9843d27f6: Status 404 returned error can't find the container with id d505fa00bd30795b6b8d4dcaff6231432149b2f8b863677ef95af1d9843d27f6
Jul 17 22:07:55.693041 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:55.693001    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4qhz6/must-gather-4lwwb" event={"ID":"f2d55ae4-0b02-4c66-a41d-8bb14dcb7307","Type":"ContainerStarted","Data":"d505fa00bd30795b6b8d4dcaff6231432149b2f8b863677ef95af1d9843d27f6"}
Jul 17 22:07:56.105922 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:56.105886    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log" limit=1024
Jul 17 22:07:56.706339 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:56.706296    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4qhz6/must-gather-4lwwb" event={"ID":"f2d55ae4-0b02-4c66-a41d-8bb14dcb7307","Type":"ContainerStarted","Data":"dbedc890794dd964e211ac06ec6c9c2e174176639b8cfaaa2cc19955061da4c8"}
Jul 17 22:07:56.706583 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:56.706356    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4qhz6/must-gather-4lwwb" event={"ID":"f2d55ae4-0b02-4c66-a41d-8bb14dcb7307","Type":"ContainerStarted","Data":"ae17c0681264bf7e324d7c76821729fc69c2bfe265a70bcd1b7c49302d5a71b9"}
Jul 17 22:07:56.721133 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:56.721076    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4qhz6/must-gather-4lwwb" podStartSLOduration=1.963456788 podStartE2EDuration="2.72103862s" totalImagesPullingTime="757.581832ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:07:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:07:56.720705419 +0000 UTC m=+3590.480272113" watchObservedRunningTime="2026-07-17 22:07:56.72103862 +0000 UTC m=+3590.480605314"
Jul 17 22:07:57.478862 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:57.478826    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6bxwh_710a2804-cc1e-49f5-8842-d392bb9aff26/global-pull-secret-syncer/0.log"
Jul 17 22:07:57.617155 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:57.617104    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7hlrp_5c629464-bbea-40de-a8d0-8d31fa115b25/konnectivity-agent/0.log"
Jul 17 22:07:57.650804 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:57.650770    2569 scope.go:117] "RemoveContainer" containerID="fa7fade868bebcb5d2d8eeacaee831f887695fe2876c953aaee6cb6d9bd9e762"
Jul 17 22:07:57.746390 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:57.746314    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-117.ec2.internal_533b658e3527087c58345fec12958890/haproxy/0.log"
Jul 17 22:07:58.734933 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:58.734844    2569 generic.go:356] "Generic (PLEG): container finished" podID="85db2a77-e6b6-4305-be81-3af9c2518ff2" containerID="588bc125d21aba174b9e3baab069bc264e6dc167b91a821918cd2063488815a5" exitCode=6
Jul 17 22:07:58.734933 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:58.734917    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj" event={"ID":"85db2a77-e6b6-4305-be81-3af9c2518ff2","Type":"ContainerDied","Data":"588bc125d21aba174b9e3baab069bc264e6dc167b91a821918cd2063488815a5"}
Jul 17 22:07:58.735519 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:58.735003    2569 scope.go:117] "RemoveContainer" containerID="fa7fade868bebcb5d2d8eeacaee831f887695fe2876c953aaee6cb6d9bd9e762"
Jul 17 22:07:58.735519 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:58.735324    2569 scope.go:117] "RemoveContainer" containerID="588bc125d21aba174b9e3baab069bc264e6dc167b91a821918cd2063488815a5"
Jul 17 22:07:58.735593 ip-10-0-140-117 kubenswrapper[2569]: E0717 22:07:58.735574    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj_odh-ai-gateway-infra(85db2a77-e6b6-4305-be81-3af9c2518ff2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj" podUID="85db2a77-e6b6-4305-be81-3af9c2518ff2"
Jul 17 22:07:59.764556 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:07:59.764462    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj"]
Jul 17 22:08:00.102742 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:00.102709    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj"
Jul 17 22:08:00.296680 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:00.296590    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85db2a77-e6b6-4305-be81-3af9c2518ff2-maas-api-service-ca\") pod \"85db2a77-e6b6-4305-be81-3af9c2518ff2\" (UID: \"85db2a77-e6b6-4305-be81-3af9c2518ff2\") "
Jul 17 22:08:00.297557 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:00.297526    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85db2a77-e6b6-4305-be81-3af9c2518ff2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "85db2a77-e6b6-4305-be81-3af9c2518ff2" (UID: "85db2a77-e6b6-4305-be81-3af9c2518ff2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:08:00.398910 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:00.398879    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85db2a77-e6b6-4305-be81-3af9c2518ff2-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:08:00.764306 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:00.764275    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj"
Jul 17 22:08:00.764552 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:00.764275    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj" event={"ID":"85db2a77-e6b6-4305-be81-3af9c2518ff2","Type":"ContainerDied","Data":"c8176633e84f751fed5775cabf7ce78c967ceaa4f1735cafb0707d113da65790"}
Jul 17 22:08:00.765038 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:00.765005    2569 scope.go:117] "RemoveContainer" containerID="588bc125d21aba174b9e3baab069bc264e6dc167b91a821918cd2063488815a5"
Jul 17 22:08:00.788243 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:00.788186    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj"]
Jul 17 22:08:00.792312 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:00.792278    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-ljblj"]
Jul 17 22:08:02.583265 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:02.583239    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2_0bd5eb95-6b65-4942-b629-5939d4c8e214/extract/0.log"
Jul 17 22:08:02.606549 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:02.606517    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2_0bd5eb95-6b65-4942-b629-5939d4c8e214/util/0.log"
Jul 17 22:08:02.627087 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:02.627053    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfzg2_0bd5eb95-6b65-4942-b629-5939d4c8e214/pull/0.log"
Jul 17 22:08:02.651796 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:02.651705    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb_c82aca7c-ff06-4c79-8c14-198701662c0d/extract/0.log"
Jul 17 22:08:02.662946 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:02.662008    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85db2a77-e6b6-4305-be81-3af9c2518ff2" path="/var/lib/kubelet/pods/85db2a77-e6b6-4305-be81-3af9c2518ff2/volumes"
Jul 17 22:08:02.673308 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:02.673225    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb_c82aca7c-ff06-4c79-8c14-198701662c0d/util/0.log"
Jul 17 22:08:02.693487 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:02.693454    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vvqjb_c82aca7c-ff06-4c79-8c14-198701662c0d/pull/0.log"
Jul 17 22:08:02.718448 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:02.718394    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b_82253133-dd19-4596-ac01-b7dd1e343483/extract/0.log"
Jul 17 22:08:02.739129 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:02.739096    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b_82253133-dd19-4596-ac01-b7dd1e343483/util/0.log"
Jul 17 22:08:02.760076 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:02.760042    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hsx2b_82253133-dd19-4596-ac01-b7dd1e343483/pull/0.log"
Jul 17 22:08:02.794649 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:02.794621    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d_3d21dbc9-6449-43d8-a58b-40e1333053ec/extract/0.log"
Jul 17 22:08:02.813329 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:02.813298    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d_3d21dbc9-6449-43d8-a58b-40e1333053ec/util/0.log"
Jul 17 22:08:02.837455 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:02.837359    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16dn9d_3d21dbc9-6449-43d8-a58b-40e1333053ec/pull/0.log"
Jul 17 22:08:03.443264 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:03.443227    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84b6ff4847-rd2kz_e390f181-7b81-4dc9-96f4-a94056ecdbdc/authorino/0.log"
Jul 17 22:08:03.479471 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:03.479417    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-2m5tx_07cffc65-fcdd-49b3-922f-c5af96a7070e/manager/0.log"
Jul 17 22:08:03.503016 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:03.502982    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-b9bnv_a7a6ffaf-914c-475b-9acc-f07241caca1d/manager/0.log"
Jul 17 22:08:03.529286 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:03.529261    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-dljpp_a09d35c4-d15f-45ad-855f-1b43f8fac1bf/kuadrant-console-plugin/0.log"
Jul 17 22:08:03.597212 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:03.597179    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-rtbw9_b856ffec-b5f2-4f2a-b3c8-80f28abb5860/registry-server/0.log"
Jul 17 22:08:03.843421 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:03.842531    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-n42th_e5248760-056f-4372-8617-4b4c22687016/manager/0.log"
Jul 17 22:08:03.864088 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:03.864041    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jb626_6b6bc014-05c4-4303-88c4-8536484d2681/limitador/0.log"
Jul 17 22:08:03.969143 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:03.968899    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-4gb9m_e4124a45-e1e0-4c94-b4f0-bf65a28e54bd/manager/0.log"
Jul 17 22:08:04.653060 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:04.653028    2569 scope.go:117] "RemoveContainer" containerID="956fd7cbb93e27c3f646034f71b3293e227bcf7d7348e1b0efac9b1cc1fe4126"
Jul 17 22:08:05.681983 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:05.681941    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-j5bcn_a29d8d5f-5087-4cd7-88b3-2d79062600bf/cluster-monitoring-operator/0.log"
Jul 17 22:08:05.823543 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:05.822009    2569 generic.go:356] "Generic (PLEG): container finished" podID="65d4c52f-6bb8-46d3-81a9-c834668f2d66" containerID="bafdbaa5b6de7e7d6e3c4110c0cdfac780a9c21e299fe8efecb3e5937eca70bf" exitCode=6
Jul 17 22:08:05.823543 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:05.822105    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb" event={"ID":"65d4c52f-6bb8-46d3-81a9-c834668f2d66","Type":"ContainerDied","Data":"bafdbaa5b6de7e7d6e3c4110c0cdfac780a9c21e299fe8efecb3e5937eca70bf"}
Jul 17 22:08:05.823543 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:05.822160    2569 scope.go:117] "RemoveContainer" containerID="956fd7cbb93e27c3f646034f71b3293e227bcf7d7348e1b0efac9b1cc1fe4126"
Jul 17 22:08:05.823543 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:05.822622    2569 scope.go:117] "RemoveContainer" containerID="bafdbaa5b6de7e7d6e3c4110c0cdfac780a9c21e299fe8efecb3e5937eca70bf"
Jul 17 22:08:05.823543 ip-10-0-140-117 kubenswrapper[2569]: E0717 22:08:05.822870    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb_odh-ai-gateway-infra(65d4c52f-6bb8-46d3-81a9-c834668f2d66)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb" podUID="65d4c52f-6bb8-46d3-81a9-c834668f2d66"
Jul 17 22:08:05.853377 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:05.853341    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-24l6g_6f04cb25-9d14-469d-83ba-f99620e0dead/node-exporter/0.log"
Jul 17 22:08:05.874765 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:05.874740    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-24l6g_6f04cb25-9d14-469d-83ba-f99620e0dead/kube-rbac-proxy/0.log"
Jul 17 22:08:05.897509 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:05.897477    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-24l6g_6f04cb25-9d14-469d-83ba-f99620e0dead/init-textfile/0.log"
Jul 17 22:08:06.851773 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:06.851714    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb"]
Jul 17 22:08:07.004882 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:07.004795    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 22:08:07.054072 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:07.054046    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 22:08:07.056538 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:07.056512    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb"
Jul 17 22:08:07.097726 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:07.097688    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65d4c52f-6bb8-46d3-81a9-c834668f2d66-maas-api-service-ca\") pod \"65d4c52f-6bb8-46d3-81a9-c834668f2d66\" (UID: \"65d4c52f-6bb8-46d3-81a9-c834668f2d66\") "
Jul 17 22:08:07.098353 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:07.098323    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d4c52f-6bb8-46d3-81a9-c834668f2d66-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "65d4c52f-6bb8-46d3-81a9-c834668f2d66" (UID: "65d4c52f-6bb8-46d3-81a9-c834668f2d66"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:08:07.199049 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:07.198958    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65d4c52f-6bb8-46d3-81a9-c834668f2d66-maas-api-service-ca\") on node \"ip-10-0-140-117.ec2.internal\" DevicePath \"\""
Jul 17 22:08:07.782452 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:07.782390    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-pgf2z_d4b360c5-c0c1-4a87-8476-85bf4b03d169/networking-console-plugin/0.log"
Jul 17 22:08:07.845119 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:07.845085    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb" event={"ID":"65d4c52f-6bb8-46d3-81a9-c834668f2d66","Type":"ContainerDied","Data":"0dea0f980306037b43d30186c5744b6e32736095fc6bf5994aeb49e958bba3dd"}
Jul 17 22:08:07.845382 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:07.845364    2569 scope.go:117] "RemoveContainer" containerID="bafdbaa5b6de7e7d6e3c4110c0cdfac780a9c21e299fe8efecb3e5937eca70bf"
Jul 17 22:08:07.845560 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:07.845103    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb"
Jul 17 22:08:07.881778 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:07.881745    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb"]
Jul 17 22:08:07.886543 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:07.886514    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-n9qzb"]
Jul 17 22:08:08.105819 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.105776    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"]
Jul 17 22:08:08.106447 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.106425    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65d4c52f-6bb8-46d3-81a9-c834668f2d66" containerName="revoke-keys"
Jul 17 22:08:08.106571 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.106450    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="65d4c52f-6bb8-46d3-81a9-c834668f2d66" containerName="revoke-keys"
Jul 17 22:08:08.106571 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.106470    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85db2a77-e6b6-4305-be81-3af9c2518ff2" containerName="revoke-keys"
Jul 17 22:08:08.106571 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.106479    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="85db2a77-e6b6-4305-be81-3af9c2518ff2" containerName="revoke-keys"
Jul 17 22:08:08.106571 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.106500    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85db2a77-e6b6-4305-be81-3af9c2518ff2" containerName="revoke-keys"
Jul 17 22:08:08.106571 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.106508    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="85db2a77-e6b6-4305-be81-3af9c2518ff2" containerName="revoke-keys"
Jul 17 22:08:08.106571 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.106519    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65d4c52f-6bb8-46d3-81a9-c834668f2d66" containerName="revoke-keys"
Jul 17 22:08:08.106571 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.106527    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="65d4c52f-6bb8-46d3-81a9-c834668f2d66" containerName="revoke-keys"
Jul 17 22:08:08.106571 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.106537    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65d4c52f-6bb8-46d3-81a9-c834668f2d66" containerName="revoke-keys"
Jul 17 22:08:08.106571 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.106546    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="65d4c52f-6bb8-46d3-81a9-c834668f2d66" containerName="revoke-keys"
Jul 17 22:08:08.106998 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.106642    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="65d4c52f-6bb8-46d3-81a9-c834668f2d66" containerName="revoke-keys"
Jul 17 22:08:08.106998 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.106657    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="65d4c52f-6bb8-46d3-81a9-c834668f2d66" containerName="revoke-keys"
Jul 17 22:08:08.106998 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.106666    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="85db2a77-e6b6-4305-be81-3af9c2518ff2" containerName="revoke-keys"
Jul 17 22:08:08.106998 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.106677    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="85db2a77-e6b6-4305-be81-3af9c2518ff2" containerName="revoke-keys"
Jul 17 22:08:08.106998 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.106686    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="65d4c52f-6bb8-46d3-81a9-c834668f2d66" containerName="revoke-keys"
Jul 17 22:08:08.114795 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.114768    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"
Jul 17 22:08:08.116986 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.116950    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4qhz6\"/\"default-dockercfg-tq2z6\""
Jul 17 22:08:08.121823 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.121787    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"]
Jul 17 22:08:08.210568 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.210531    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6f63982-eb33-46b4-8b76-d0577b38dca1-sys\") pod \"perf-node-gather-daemonset-bdthq\" (UID: \"d6f63982-eb33-46b4-8b76-d0577b38dca1\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"
Jul 17 22:08:08.210759 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.210599    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d6f63982-eb33-46b4-8b76-d0577b38dca1-proc\") pod \"perf-node-gather-daemonset-bdthq\" (UID: \"d6f63982-eb33-46b4-8b76-d0577b38dca1\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"
Jul 17 22:08:08.210759 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.210640    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d6f63982-eb33-46b4-8b76-d0577b38dca1-podres\") pod \"perf-node-gather-daemonset-bdthq\" (UID: \"d6f63982-eb33-46b4-8b76-d0577b38dca1\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"
Jul 17 22:08:08.210759 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.210724    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kd4h\" (UniqueName: \"kubernetes.io/projected/d6f63982-eb33-46b4-8b76-d0577b38dca1-kube-api-access-7kd4h\") pod \"perf-node-gather-daemonset-bdthq\" (UID: \"d6f63982-eb33-46b4-8b76-d0577b38dca1\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"
Jul 17 22:08:08.210875 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.210782    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6f63982-eb33-46b4-8b76-d0577b38dca1-lib-modules\") pod \"perf-node-gather-daemonset-bdthq\" (UID: \"d6f63982-eb33-46b4-8b76-d0577b38dca1\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"
Jul 17 22:08:08.311912 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.311867    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d6f63982-eb33-46b4-8b76-d0577b38dca1-proc\") pod \"perf-node-gather-daemonset-bdthq\" (UID: \"d6f63982-eb33-46b4-8b76-d0577b38dca1\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"
Jul 17 22:08:08.312107 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.311922    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d6f63982-eb33-46b4-8b76-d0577b38dca1-podres\") pod \"perf-node-gather-daemonset-bdthq\" (UID: \"d6f63982-eb33-46b4-8b76-d0577b38dca1\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"
Jul 17 22:08:08.312107 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.311993    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kd4h\" (UniqueName: \"kubernetes.io/projected/d6f63982-eb33-46b4-8b76-d0577b38dca1-kube-api-access-7kd4h\") pod \"perf-node-gather-daemonset-bdthq\" (UID: \"d6f63982-eb33-46b4-8b76-d0577b38dca1\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"
Jul 17 22:08:08.312107 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.312011    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d6f63982-eb33-46b4-8b76-d0577b38dca1-proc\") pod \"perf-node-gather-daemonset-bdthq\" (UID: \"d6f63982-eb33-46b4-8b76-d0577b38dca1\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"
Jul 17 22:08:08.312107 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.312050    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6f63982-eb33-46b4-8b76-d0577b38dca1-lib-modules\") pod \"perf-node-gather-daemonset-bdthq\" (UID: \"d6f63982-eb33-46b4-8b76-d0577b38dca1\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"
Jul 17 22:08:08.312740 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.312134    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6f63982-eb33-46b4-8b76-d0577b38dca1-sys\") pod \"perf-node-gather-daemonset-bdthq\" (UID: \"d6f63982-eb33-46b4-8b76-d0577b38dca1\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"
Jul 17 22:08:08.312740 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.312143    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d6f63982-eb33-46b4-8b76-d0577b38dca1-podres\") pod \"perf-node-gather-daemonset-bdthq\" (UID: \"d6f63982-eb33-46b4-8b76-d0577b38dca1\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"
Jul 17 22:08:08.312740 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.312239    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6f63982-eb33-46b4-8b76-d0577b38dca1-sys\") pod \"perf-node-gather-daemonset-bdthq\" (UID: \"d6f63982-eb33-46b4-8b76-d0577b38dca1\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"
Jul 17 22:08:08.312740 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.312321    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6f63982-eb33-46b4-8b76-d0577b38dca1-lib-modules\") pod \"perf-node-gather-daemonset-bdthq\" (UID: \"d6f63982-eb33-46b4-8b76-d0577b38dca1\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"
Jul 17 22:08:08.320544 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.320492    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kd4h\" (UniqueName: \"kubernetes.io/projected/d6f63982-eb33-46b4-8b76-d0577b38dca1-kube-api-access-7kd4h\") pod \"perf-node-gather-daemonset-bdthq\" (UID: \"d6f63982-eb33-46b4-8b76-d0577b38dca1\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"
Jul 17 22:08:08.426168 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.426086    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/1.log"
Jul 17 22:08:08.433980 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.433950    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"
Jul 17 22:08:08.435520 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.435497    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-k7c4s_c0b7933f-7894-4ea9-8cbc-3704888fdeed/console-operator/2.log"
Jul 17 22:08:08.615949 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.615920    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"]
Jul 17 22:08:08.617110 ip-10-0-140-117 kubenswrapper[2569]: W0717 22:08:08.617074    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd6f63982_eb33_46b4_8b76_d0577b38dca1.slice/crio-5f39c841db37acb79687b5ad67ae540ebce7f02d2411fac5b10af3ff1a1ba456 WatchSource:0}: Error finding container 5f39c841db37acb79687b5ad67ae540ebce7f02d2411fac5b10af3ff1a1ba456: Status 404 returned error can't find the container with id 5f39c841db37acb79687b5ad67ae540ebce7f02d2411fac5b10af3ff1a1ba456
Jul 17 22:08:08.658454 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.658419    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65d4c52f-6bb8-46d3-81a9-c834668f2d66" path="/var/lib/kubelet/pods/65d4c52f-6bb8-46d3-81a9-c834668f2d66/volumes"
Jul 17 22:08:08.858671 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.858630    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq" event={"ID":"d6f63982-eb33-46b4-8b76-d0577b38dca1","Type":"ContainerStarted","Data":"89e63fef4f44f350ea243b34cd91af02ba1d83e005bb1f7ecabea033bf05b78e"}
Jul 17 22:08:08.858859 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.858697    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq" event={"ID":"d6f63982-eb33-46b4-8b76-d0577b38dca1","Type":"ContainerStarted","Data":"5f39c841db37acb79687b5ad67ae540ebce7f02d2411fac5b10af3ff1a1ba456"}
Jul 17 22:08:08.859542 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.859512    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"
Jul 17 22:08:08.874776 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.874731    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq" podStartSLOduration=0.874717556 podStartE2EDuration="874.717556ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 22:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:08:08.873952384 +0000 UTC m=+3602.633519076" watchObservedRunningTime="2026-07-17 22:08:08.874717556 +0000 UTC m=+3602.634284249"
Jul 17 22:08:08.983575 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:08.983544    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-54vt8_92af7adf-f9b6-4f1b-9ceb-3a915e7cb85e/download-server/0.log"
Jul 17 22:08:09.561964 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:09.561935    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-gw6z9_05369579-52bc-4e86-9fe2-32d608748a6d/volume-data-source-validator/0.log"
Jul 17 22:08:10.661170 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:10.661144    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bjqvd_8859418c-d1a7-4728-989b-04addc542331/dns/0.log"
Jul 17 22:08:10.679140 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:10.679110    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bjqvd_8859418c-d1a7-4728-989b-04addc542331/kube-rbac-proxy/0.log"
Jul 17 22:08:10.699734 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:10.699705    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6fvtz_98470465-ed0c-4896-8871-e8d9354e3dfa/dns-node-resolver/0.log"
Jul 17 22:08:11.265700 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:11.265660    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7556ff4784-ntbm5_f1ccff32-42e3-4ce5-a863-911e4681f43f/registry/0.log"
Jul 17 22:08:11.337251 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:11.337219    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qn5pr_a9d9e563-636a-4912-8557-a9d71096efb8/node-ca/0.log"
Jul 17 22:08:12.410158 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:12.410113    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cf476465c-wbzp5_efda4d41-9563-4e36-b789-859c4786e297/kube-auth-proxy/0.log"
Jul 17 22:08:12.550682 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:12.550652    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-756f96dd8-2m7s2_2cfce8d6-1da4-4003-8183-31744592793b/router/0.log"
Jul 17 22:08:13.229375 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:13.229322    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-97qxw_3ca96db1-3372-483e-9e19-2fb5046e81bc/serve-healthcheck-canary/0.log"
Jul 17 22:08:13.730808 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:13.730774    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-jgv9v_6878c658-1dc7-4cd2-811f-ec6a726f07ee/insights-operator/0.log"
Jul 17 22:08:13.735540 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:13.735504    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-jgv9v_6878c658-1dc7-4cd2-811f-ec6a726f07ee/insights-operator/1.log"
Jul 17 22:08:13.880324 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:13.880285    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zwjfb_6883c34d-3012-4a25-90d7-df75eeaf87b3/kube-rbac-proxy/0.log"
Jul 17 22:08:13.897815 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:13.897760    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zwjfb_6883c34d-3012-4a25-90d7-df75eeaf87b3/exporter/0.log"
Jul 17 22:08:13.918681 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:13.918649    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zwjfb_6883c34d-3012-4a25-90d7-df75eeaf87b3/extractor/0.log"
Jul 17 22:08:15.884910 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:15.884832    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-bdthq"
Jul 17 22:08:15.975262 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:15.975221    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-swflf_7134f6e3-e6cc-48f0-85c8-87dd703137d8/manager/0.log"
Jul 17 22:08:16.251674 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:16.251493    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-l4wzj_e4c502b9-5cc5-40d3-ac93-d52a0f629279/manager/0.log"
Jul 17 22:08:16.375993 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:16.375964    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-lms8h_2106c72f-6b22-4c1b-8328-9fb93b711349/server/0.log"
Jul 17 22:08:16.483933 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:16.483872    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-r8sf5_aaf6b929-c2d5-48fb-ab6b-535386eb9635/manager/0.log"
Jul 17 22:08:16.600110 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:16.600077    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b8f6b79df-jqdz6_abe9dd49-3a15-4f58-b19f-e871479de840/manager/0.log"
Jul 17 22:08:16.619623 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:16.619593    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b8f6b79df-jqdz6_abe9dd49-3a15-4f58-b19f-e871479de840/copy-manifests/0.log"
Jul 17 22:08:18.861297 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:18.861262    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cf476465c-wbzp5_efda4d41-9563-4e36-b789-859c4786e297/kube-auth-proxy/0.log"
Jul 17 22:08:18.962292 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:18.962236    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-756f96dd8-2m7s2_2cfce8d6-1da4-4003-8183-31744592793b/router/0.log"
Jul 17 22:08:20.288786 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:20.288671    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cf476465c-wbzp5_efda4d41-9563-4e36-b789-859c4786e297/kube-auth-proxy/0.log"
Jul 17 22:08:20.403482 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:20.403451    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-756f96dd8-2m7s2_2cfce8d6-1da4-4003-8183-31744592793b/router/0.log"
Jul 17 22:08:23.245426 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:23.245376    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-z4l8p_61b45b94-534a-419f-a4a3-424b6d8cee2a/migrator/0.log"
Jul 17 22:08:23.263914 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:23.263885    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-z4l8p_61b45b94-534a-419f-a4a3-424b6d8cee2a/graceful-termination/0.log"
Jul 17 22:08:23.605049 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:23.605013    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-9szm7_f596f331-2b59-4782-b6cc-ff757d3ff94c/kube-storage-version-migrator-operator/1.log"
Jul 17 22:08:23.606712 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:23.606689    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-9szm7_f596f331-2b59-4782-b6cc-ff757d3ff94c/kube-storage-version-migrator-operator/0.log"
Jul 17 22:08:24.969144 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:24.969117    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hxjhv_9dc91e48-32c6-42ad-b3e9-7be1ec357d6f/kube-multus-additional-cni-plugins/0.log"
Jul 17 22:08:24.992034 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:24.992007    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hxjhv_9dc91e48-32c6-42ad-b3e9-7be1ec357d6f/egress-router-binary-copy/0.log"
Jul 17 22:08:25.009993 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:25.009969    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hxjhv_9dc91e48-32c6-42ad-b3e9-7be1ec357d6f/cni-plugins/0.log"
Jul 17 22:08:25.027476 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:25.027452    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hxjhv_9dc91e48-32c6-42ad-b3e9-7be1ec357d6f/bond-cni-plugin/0.log"
Jul 17 22:08:25.045020 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:25.044991    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hxjhv_9dc91e48-32c6-42ad-b3e9-7be1ec357d6f/routeoverride-cni/0.log"
Jul 17 22:08:25.065682 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:25.065656    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hxjhv_9dc91e48-32c6-42ad-b3e9-7be1ec357d6f/whereabouts-cni-bincopy/0.log"
Jul 17 22:08:25.083704 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:25.083668    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hxjhv_9dc91e48-32c6-42ad-b3e9-7be1ec357d6f/whereabouts-cni/0.log"
Jul 17 22:08:25.152294 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:25.152268    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-djkwb_867f0a7f-0397-4df5-8832-c6ee935d00a3/kube-multus/0.log"
Jul 17 22:08:25.308691 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:25.308598    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hnhk2_89bdc455-1935-456b-bc58-288a0ed415a0/network-metrics-daemon/0.log"
Jul 17 22:08:25.322201 ip-10-0-140-117 kubenswrapper[2569]: I0717 22:08:25.322162    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hnhk2_89bdc455-1935-456b-bc58-288a0ed415a0/kube-rbac-proxy/0.log"