Jul 22 02:18:50.448880 ip-10-0-140-200 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 22 02:18:50.448891 ip-10-0-140-200 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 22 02:18:50.448898 ip-10-0-140-200 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 22 02:18:50.449111 ip-10-0-140-200 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 22 02:19:00.221928 ip-10-0-140-200 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Jul 22 02:19:00.221943 ip-10-0-140-200 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d0ad850eae2c4e8ab7df7a03dac9cbeb --
Jul 22 02:20:47.545252 ip-10-0-140-200 systemd[1]: Starting Kubernetes Kubelet...
Jul 22 02:20:48.053258 ip-10-0-140-200 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 22 02:20:48.053258 ip-10-0-140-200 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 22 02:20:48.053258 ip-10-0-140-200 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 22 02:20:48.053258 ip-10-0-140-200 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 22 02:20:48.053258 ip-10-0-140-200 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 22 02:20:48.053258 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.013359    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 22 02:20:48.053258 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020227    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 02:20:48.053258 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020241    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 02:20:48.053258 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020246    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 02:20:48.053258 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020249    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 02:20:48.053258 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020252    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 02:20:48.053258 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020255    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 02:20:48.053258 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020258    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 02:20:48.053258 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020260    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 02:20:48.054073 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020263    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 02:20:48.054073 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020266    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 02:20:48.054073 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020268    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 02:20:48.054073 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020271    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 02:20:48.054073 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020273    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 02:20:48.054073 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020276    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 02:20:48.054073 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020279    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 02:20:48.054073 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020281    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 02:20:48.054073 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020284    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 02:20:48.054073 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020287    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 02:20:48.054073 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020289    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 02:20:48.054073 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020299    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 02:20:48.054073 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020303    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 02:20:48.054073 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020306    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 02:20:48.054073 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020308    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 02:20:48.054073 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020311    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 02:20:48.054073 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020314    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 02:20:48.054073 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020325    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 02:20:48.054073 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020328    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 02:20:48.054073 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020330    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 02:20:48.054673 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020333    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 02:20:48.054673 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020335    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 02:20:48.054673 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020338    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 02:20:48.054673 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020340    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 02:20:48.054673 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020344    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 02:20:48.054673 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020346    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 02:20:48.054673 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020350    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 02:20:48.054673 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020352    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 02:20:48.054673 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020355    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 02:20:48.054673 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020357    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 02:20:48.054673 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020360    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 02:20:48.054673 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020362    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 02:20:48.054673 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020365    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 02:20:48.054673 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020368    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 02:20:48.054673 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020370    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 02:20:48.054673 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020373    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 02:20:48.054673 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020375    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 02:20:48.054673 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020378    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 02:20:48.054673 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020381    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 02:20:48.054673 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020383    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 02:20:48.055261 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020386    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 02:20:48.055261 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020390    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 02:20:48.055261 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020398    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 02:20:48.055261 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020401    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 02:20:48.055261 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020403    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 02:20:48.055261 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020406    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 02:20:48.055261 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020409    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 02:20:48.055261 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020411    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 02:20:48.055261 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020414    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 02:20:48.055261 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020416    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 02:20:48.055261 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020419    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 02:20:48.055261 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020421    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 02:20:48.055261 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020424    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 02:20:48.055261 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020426    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 02:20:48.055261 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020429    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 02:20:48.055261 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020432    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 02:20:48.055261 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020435    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 02:20:48.055261 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020438    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 02:20:48.055261 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020441    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 02:20:48.055261 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020443    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 02:20:48.055818 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020446    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 02:20:48.055818 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020449    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 02:20:48.055818 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020451    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 02:20:48.055818 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020454    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 02:20:48.055818 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020456    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 02:20:48.055818 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020458    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 02:20:48.055818 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020461    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 02:20:48.055818 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020463    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 02:20:48.055818 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020467    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 02:20:48.055818 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020471    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 02:20:48.055818 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020475    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 02:20:48.055818 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020478    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 02:20:48.055818 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020480    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 02:20:48.055818 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020482    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 02:20:48.055818 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020487    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 02:20:48.055818 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020497    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 02:20:48.055818 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020500    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 02:20:48.055818 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020503    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 02:20:48.055818 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020507    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 02:20:48.056335 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020510    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 02:20:48.056335 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020536    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 02:20:48.056335 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020540    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 02:20:48.056335 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020543    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 02:20:48.056335 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020546    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 02:20:48.056335 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020548    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 02:20:48.056335 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020551    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 02:20:48.056335 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020555    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 02:20:48.056335 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020558    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 02:20:48.056335 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020561    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 02:20:48.056335 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020564    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 02:20:48.056335 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.020567    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 02:20:48.056335 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021307    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 02:20:48.056335 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021315    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 02:20:48.056335 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021318    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 02:20:48.056335 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021321    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 02:20:48.056335 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021324    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 02:20:48.056335 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021326    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 02:20:48.056335 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021329    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 02:20:48.056335 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021332    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 02:20:48.056933 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021335    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 02:20:48.056933 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021337    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 02:20:48.056933 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021340    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 02:20:48.056933 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021343    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 02:20:48.056933 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021345    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 02:20:48.056933 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021348    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 02:20:48.056933 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021352    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 02:20:48.056933 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021356    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 02:20:48.056933 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021358    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 02:20:48.056933 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021362    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 02:20:48.056933 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021365    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 02:20:48.056933 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021368    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 02:20:48.056933 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021370    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 02:20:48.056933 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021372    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 02:20:48.056933 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021375    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 02:20:48.056933 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021378    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 02:20:48.056933 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021380    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 02:20:48.056933 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021383    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 02:20:48.056933 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021385    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 02:20:48.057603 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021388    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 02:20:48.057603 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021392    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 02:20:48.057603 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021395    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 02:20:48.057603 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021397    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 02:20:48.057603 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021400    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 02:20:48.057603 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021402    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 02:20:48.057603 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021406    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 02:20:48.057603 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021411    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 02:20:48.057603 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021414    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 02:20:48.057603 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021417    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 02:20:48.057603 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021420    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 02:20:48.057603 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021423    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 02:20:48.057603 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021425    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 02:20:48.057603 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021428    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 02:20:48.057603 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021432    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 02:20:48.057603 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021435    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 02:20:48.057603 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021438    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 02:20:48.057603 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021441    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 02:20:48.057603 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021444    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 02:20:48.058151 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021447    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 02:20:48.058151 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021449    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 02:20:48.058151 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021452    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 02:20:48.058151 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021455    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 02:20:48.058151 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021458    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 02:20:48.058151 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021460    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 02:20:48.058151 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021463    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 02:20:48.058151 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021465    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 02:20:48.058151 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021468    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 02:20:48.058151 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021470    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 02:20:48.058151 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021473    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 02:20:48.058151 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021475    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 02:20:48.058151 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021478    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 02:20:48.058151 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021480    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 02:20:48.058151 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021483    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 02:20:48.058151 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021486    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 02:20:48.058151 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021489    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 02:20:48.058151 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021491    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 02:20:48.058151 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021494    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021496    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021499    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021502    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021504    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021508    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021511    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021513    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021516    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021518    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021521    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021524    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021526    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021529    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021532    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021534    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021537    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021539    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021542    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021545    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021548    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 02:20:48.058661 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021550    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 02:20:48.059259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021553    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 02:20:48.059259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021555    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 02:20:48.059259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021558    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 02:20:48.059259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021560    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 02:20:48.059259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021563    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 02:20:48.059259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021566    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 02:20:48.059259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021568    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 02:20:48.059259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021571    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 02:20:48.059259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021573    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 02:20:48.059259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021576    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 02:20:48.059259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021578    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 02:20:48.059259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021581    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 02:20:48.059259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021583    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 02:20:48.059259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021952    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 02:20:48.059259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021975    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 02:20:48.059259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021980    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 02:20:48.059259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021983    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 02:20:48.059259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021987    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 02:20:48.059259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021990    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 02:20:48.059829 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021993    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 02:20:48.059829 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021996    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 02:20:48.059829 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.021999    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 02:20:48.059829 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022001    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 02:20:48.059829 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022004    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 02:20:48.059829 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022007    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 02:20:48.059829 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022010    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 02:20:48.059829 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022013    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 02:20:48.059829 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022015    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 02:20:48.059829 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022018    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 02:20:48.059829 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022021    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 02:20:48.059829 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022024    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 02:20:48.059829 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022027    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 02:20:48.059829 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022030    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 02:20:48.059829 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022032    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 02:20:48.059829 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022035    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 02:20:48.059829 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022037    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 02:20:48.059829 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022040    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 02:20:48.059829 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022043    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 02:20:48.059829 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022045    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 02:20:48.060443 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022048    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 02:20:48.060443 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022051    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 02:20:48.060443 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022053    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 02:20:48.060443 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022056    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 02:20:48.060443 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022058    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 02:20:48.060443 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022060    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 02:20:48.060443 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022063    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 02:20:48.060443 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022065    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 02:20:48.060443 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022068    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 02:20:48.060443 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022070    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 02:20:48.060443 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022073    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 02:20:48.060443 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022075    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 02:20:48.060443 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022078    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 02:20:48.060443 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022080    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 02:20:48.060443 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022083    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 02:20:48.060443 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022085    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 02:20:48.060443 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022087    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 02:20:48.060443 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022090    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 02:20:48.060443 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022092    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 02:20:48.061012 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022095    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 02:20:48.061012 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022097    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 02:20:48.061012 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022100    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 02:20:48.061012 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022102    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 02:20:48.061012 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022106    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 02:20:48.061012 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022110    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 02:20:48.061012 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022113    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 02:20:48.061012 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022116    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 02:20:48.061012 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022119    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 02:20:48.061012 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022122    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 02:20:48.061012 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022125    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 02:20:48.061012 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022127    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 02:20:48.061012 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022129    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 02:20:48.061012 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022132    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 02:20:48.061012 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022134    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 02:20:48.061012 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022138    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 02:20:48.061012 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022141    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 02:20:48.061012 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022144    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 02:20:48.061012 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022147    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 02:20:48.061594 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022150    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 02:20:48.061594 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022153    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 02:20:48.061594 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022156    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 02:20:48.061594 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022158    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 02:20:48.061594 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022161    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 02:20:48.061594 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022164    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 02:20:48.061594 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022166    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 02:20:48.061594 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022169    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 02:20:48.061594 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022171    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 02:20:48.061594 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022175    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 02:20:48.061594 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022180    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 02:20:48.061594 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022183    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 02:20:48.061594 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022186    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 02:20:48.061594 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022188    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 02:20:48.061594 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022191    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 02:20:48.061594 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022193    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 02:20:48.061594 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022196    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 02:20:48.061594 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022198    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 02:20:48.061594 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022205    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 02:20:48.061594 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022208    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022211    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022213    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022216    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022219    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022222    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022224    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022227    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022230    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022232    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022235    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022238    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022240    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022243    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022245    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022248    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022318    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022325    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022331    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022335    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022339    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 22 02:20:48.062341 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022342    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022347    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022352    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022355    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022358    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022362    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022365    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022368    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022371    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022374    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022377    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022380    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022383    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022388    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022391    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022394    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022398    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022401    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022405    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022408    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022411    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022414    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022417    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022420    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 22 02:20:48.063152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022423    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022426    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022429    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022434    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022437    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022440    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022442    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022447    2565 flags.go:64] FLAG: --enable-server="true"
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022450    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022454    2565 flags.go:64] FLAG: --event-burst="100"
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022458    2565 flags.go:64] FLAG: --event-qps="50"
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022461    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022464    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022467    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022471    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022474    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022477    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022480    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022483    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022486    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022489    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022492    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022495    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022498    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022501    2565 flags.go:64] FLAG: --feature-gates=""
Jul 22 02:20:49.676727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022504    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 22 02:20:48.087678 ip-10-0-140-200 systemd[1]: Started Kubernetes Kubelet.
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022507    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022510    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022513    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022516    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022519    2565 flags.go:64] FLAG: --help="false"
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022522    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-140-200.ec2.internal"
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022525    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022528    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022531    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022534    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022537    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022540    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022543    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022547    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022550    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022553    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022559    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022561    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022564    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022567    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022570    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022573    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022576    2565 flags.go:64] FLAG: --lock-file=""
Jul 22 02:20:49.693684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022579    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022581    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022584    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022590    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022593    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022596    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022598    2565 flags.go:64] FLAG: --logging-format="text"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022601    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022605    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022607    2565 flags.go:64] FLAG: --manifest-url=""
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022610    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022615    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022618    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022621    2565 flags.go:64] FLAG: --max-pods="110"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022624    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022627    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022630    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022633    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022636    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022639    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022642    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022649    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022654    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022657    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 22 02:20:49.695537 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022660    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022664    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022669    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022672    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022675    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022678    2565 flags.go:64] FLAG: --port="10250"
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022681    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022684    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0aff5cad11b6c2868"
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022687    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022690    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022693    2565 flags.go:64] FLAG: --register-node="true"
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022695    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022699    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022703    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022706    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022708    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022712    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022715    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022717    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022720    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022723    2565 flags.go:64] FLAG: --runonce="false"
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022725    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022728    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022731    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022734    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 22 02:20:49.696892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022737    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022740    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022743    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022746    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022748    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022753    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022755    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022758    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022762    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022766    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022769    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022774    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022777    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022779    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022783    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022786    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022789    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022792    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022795    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022798    2565 flags.go:64] FLAG: --v="2"
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022802    2565 flags.go:64] FLAG: --version="false"
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022806    2565 flags.go:64] FLAG: --vmodule=""
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022810    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.022813    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022921    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 02:20:49.698938 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022925    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 02:20:49.700193 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022928    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 02:20:49.700193 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022931    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 02:20:49.700193 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022933    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 02:20:49.700193 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022936    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 02:20:49.700193 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022939    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 02:20:49.700193 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022941    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 02:20:49.700193 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022944    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 02:20:49.700193 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022947    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 02:20:49.700193 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022949    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 02:20:49.700193 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022952    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 02:20:49.700193 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022970    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 02:20:49.700193 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022975    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 02:20:49.700193 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022978    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 02:20:49.700193 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022981    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 02:20:49.700193 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022983    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 02:20:49.700193 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022987    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 02:20:49.700193 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022990    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 02:20:49.700193 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022993    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 02:20:49.700193 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022995    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 02:20:49.700193 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.022998    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 02:20:49.701259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023001    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 02:20:49.701259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023003    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 02:20:49.701259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023006    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 02:20:49.701259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023010    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 02:20:49.701259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023014    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 02:20:49.701259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023017    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 02:20:49.701259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023020    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 02:20:49.701259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023022    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 02:20:49.701259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023026    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 02:20:49.701259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023028    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 02:20:49.701259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023031    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 02:20:49.701259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023034    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 02:20:49.701259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023036    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 02:20:49.701259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023039    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 02:20:49.701259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023041    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 02:20:49.701259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023044    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 02:20:49.701259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023047    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 02:20:49.701259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023049    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 02:20:49.701259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023052    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 02:20:49.701259 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023054    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 02:20:49.702378 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023057    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 02:20:49.702378 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023059    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 02:20:49.702378 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023062    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 02:20:49.702378 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023068    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 02:20:49.702378 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023070    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 02:20:49.702378 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023073    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 02:20:49.702378 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023076    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 02:20:49.702378 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023079    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 02:20:49.702378 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023083    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 02:20:49.702378 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023087    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 02:20:49.702378 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023090    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 02:20:49.702378 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023093    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 02:20:49.702378 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023095    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 02:20:49.702378 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023098    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 02:20:49.702378 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023101    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 02:20:49.702378 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023103    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 02:20:49.702378 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023106    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 02:20:49.702378 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023108    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 02:20:49.702378 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023111    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 02:20:49.703779 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023114    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 02:20:49.703779 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023116    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 02:20:49.703779 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023119    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 02:20:49.703779 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023140    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 02:20:49.703779 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023144    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 02:20:49.703779 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023147    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 02:20:49.703779 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023151    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 02:20:49.703779 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023153    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 02:20:49.703779 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023157    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 02:20:49.703779 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023161    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 02:20:49.703779 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023164    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 02:20:49.703779 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023167    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 02:20:49.703779 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023169    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 02:20:49.703779 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023172    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 02:20:49.703779 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023175    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 02:20:49.703779 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023178    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 02:20:49.703779 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023183    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 02:20:49.703779 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023186    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 02:20:49.703779 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023188    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 02:20:49.704842 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023191    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 02:20:49.704842 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023195    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 02:20:49.704842 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023197    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 02:20:49.704842 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023200    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 02:20:49.704842 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023203    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 02:20:49.704842 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023205    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 02:20:49.704842 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023208    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 02:20:49.704842 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023211    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 02:20:49.704842 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023213    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 02:20:49.704842 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023216    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 02:20:49.704842 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023219    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 02:20:49.704842 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023222    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 02:20:49.704842 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023225    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 02:20:49.704842 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023228    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 02:20:49.704842 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023230    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 02:20:49.704842 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023233    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 02:20:49.704842 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023236    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 02:20:49.704842 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023239    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 02:20:49.704842 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.023242    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 02:20:49.706294 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.023250    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 22 02:20:49.706294 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.030600    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 22 02:20:49.706294 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.030614    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 22 02:20:49.706294 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.031532    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 22 02:20:49.706294 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.031539    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 22 02:20:49.706294 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031594    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 02:20:49.706294 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031601    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 02:20:49.706294 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031605    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 02:20:49.706294 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031608    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 02:20:49.706294 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031612    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 02:20:49.706294 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031616    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 02:20:49.706294 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031619    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 02:20:49.706294 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031621    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 02:20:49.706294 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031624    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 02:20:49.707167 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031627    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 02:20:49.707167 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031630    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 02:20:49.707167 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031633    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 02:20:49.707167 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031636    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 02:20:49.707167 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031639    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 02:20:49.707167 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031642    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 02:20:49.707167 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031645    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 02:20:49.707167 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031647    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 02:20:49.707167 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031650    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 02:20:49.707167 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031653    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 02:20:49.707167 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031655    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 02:20:49.707167 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031658    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 02:20:49.707167 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031661    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 02:20:49.707167 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031663    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 02:20:49.707167 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031666    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 02:20:49.707167 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031669    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 02:20:49.707167 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031672    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 02:20:49.707167 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031675    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 02:20:49.707167 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031678    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 02:20:49.707167 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031680    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 02:20:49.708639 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031683    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 02:20:49.708639 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031686    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 02:20:49.708639 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031688    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 02:20:49.708639 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031691    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 02:20:49.708639 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031694    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 02:20:49.708639 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031696    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 02:20:49.708639 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031699    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 02:20:49.708639 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031701    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 02:20:49.708639 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031704    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 02:20:49.708639 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031707    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 02:20:49.708639 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031710    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 02:20:49.708639 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031713    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 02:20:49.708639 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031716    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 02:20:49.708639 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031718    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 02:20:49.708639 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031721    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 02:20:49.708639 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031723    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 02:20:49.708639 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031727    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 02:20:49.708639 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031730    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 02:20:49.708639 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031732    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 02:20:49.709626 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031735    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 02:20:49.709626 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031737    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 02:20:49.709626 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031740    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 02:20:49.709626 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031743    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 02:20:49.709626 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031745    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 02:20:49.709626 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031748    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 02:20:49.709626 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031751    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 02:20:49.709626 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031754    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 02:20:49.709626 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031756    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 02:20:49.709626 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031759    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 02:20:49.709626 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031762    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 02:20:49.709626 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031765    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 02:20:49.709626 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031767    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 02:20:49.709626 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031770    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 02:20:49.709626 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031774    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 02:20:49.709626 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031776    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 02:20:49.709626 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031779    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 02:20:49.709626 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031782    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 02:20:49.709626 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031784    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 02:20:49.711116 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031787    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 02:20:49.711116 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031789    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 02:20:49.711116 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031792    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 02:20:49.711116 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031794    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 02:20:49.711116 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031796    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 02:20:49.711116 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031799    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 02:20:49.711116 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031801    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 02:20:49.711116 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031804    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 02:20:49.711116 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031807    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 02:20:49.711116 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031810    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 02:20:49.711116 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031814    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 02:20:49.711116 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031817    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 02:20:49.711116 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031820    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 02:20:49.711116 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031822    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 02:20:49.711116 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031825    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 02:20:49.711116 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031827    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 02:20:49.711116 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031830    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 02:20:49.711116 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031833    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 02:20:49.711116 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031837    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 02:20:49.711116 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031839    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 02:20:49.711970 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031842    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 02:20:49.711970 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031845    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 02:20:49.711970 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031847    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 02:20:49.711970 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031850    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 02:20:49.711970 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031852    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 02:20:49.711970 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031855    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 02:20:49.711970 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031858    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 02:20:49.711970 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031861    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 02:20:49.711970 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031864    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 02:20:49.711970 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031866    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 02:20:49.711970 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031869    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 02:20:49.711970 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.031872    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 02:20:49.711970 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.031877    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 22 02:20:49.711970 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032027    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 02:20:49.711970 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032033    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 02:20:49.713393 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032036    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 02:20:49.713393 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032039    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 02:20:49.713393 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032042    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 02:20:49.713393 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032044    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 02:20:49.713393 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032047    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 02:20:49.713393 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032049    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 02:20:49.713393 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032052    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 02:20:49.713393 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032057    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 02:20:49.713393 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032059    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 02:20:49.713393 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032062    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 02:20:49.713393 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032065    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 02:20:49.713393 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032067    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 02:20:49.713393 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032070    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 02:20:49.713393 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032073    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 02:20:49.713393 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032075    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 02:20:49.713393 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032077    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 02:20:49.713393 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032081    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 02:20:49.713393 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032084    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 02:20:49.713393 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032087    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 02:20:49.713393 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032089    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 02:20:49.713996 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032092    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 02:20:49.713996 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032094    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 02:20:49.713996 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032097    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 02:20:49.713996 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032100    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 02:20:49.713996 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032102    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 02:20:49.713996 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032105    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 02:20:49.713996 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032108    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 02:20:49.713996 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032111    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 02:20:49.713996 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032113    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 02:20:49.713996 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032116    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 02:20:49.713996 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032119    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 02:20:49.713996 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032121    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 02:20:49.713996 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032124    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 02:20:49.713996 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032126    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 02:20:49.713996 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032129    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 02:20:49.713996 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032131    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 02:20:49.713996 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032134    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 02:20:49.713996 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032137    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 02:20:49.713996 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032139    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 02:20:49.713996 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032143    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 02:20:49.714570 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032146    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 02:20:49.714570 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032148    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 02:20:49.714570 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032150    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 02:20:49.714570 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032153    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 02:20:49.714570 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032155    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 02:20:49.714570 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032158    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 02:20:49.714570 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032160    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 02:20:49.714570 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032163    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 02:20:49.714570 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032167    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 02:20:49.714570 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032170    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 02:20:49.714570 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032172    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 02:20:49.714570 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032175    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 02:20:49.714570 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032178    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 02:20:49.714570 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032180    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 02:20:49.714570 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032183    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 02:20:49.714570 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032185    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 02:20:49.714570 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032188    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 02:20:49.714570 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032190    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 02:20:49.714570 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032193    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 02:20:49.714570 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032196    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 02:20:49.715138 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032198    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 02:20:49.715138 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032201    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 02:20:49.715138 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032204    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 02:20:49.715138 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032207    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 02:20:49.715138 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032211    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 02:20:49.715138 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032213    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 02:20:49.715138 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032216    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 02:20:49.715138 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032220    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 02:20:49.715138 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032223    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 02:20:49.715138 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032226    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 02:20:49.715138 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032229    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 02:20:49.715138 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032233    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 02:20:49.715138 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032235    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 02:20:49.715138 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032238    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 02:20:49.715138 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032241    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 02:20:49.715138 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032244    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 02:20:49.715138 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032247    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 02:20:49.715138 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032249    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 02:20:49.715138 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032252    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 02:20:49.715642 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032254    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 02:20:49.715642 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032259    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 02:20:49.715642 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032263    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 02:20:49.715642 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032266    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 02:20:49.715642 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032269    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 02:20:49.715642 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032272    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 02:20:49.715642 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032274    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 02:20:49.715642 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032277    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 02:20:49.715642 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032280    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 02:20:49.715642 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032283    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 02:20:49.715642 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032286    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 02:20:49.715642 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032289    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 02:20:49.715642 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032300    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 02:20:49.715642 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032305    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 02:20:49.715642 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032307    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 02:20:49.715642 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032310    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 02:20:49.715642 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032313    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 02:20:49.715642 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:48.032316    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 02:20:49.716175 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.032320    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 22 02:20:49.716175 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.032425    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 22 02:20:49.716175 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.034270    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 22 02:20:49.716175 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.035635    2565 server.go:1019] "Starting client certificate rotation"
Jul 22 02:20:49.716175 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.035735    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 22 02:20:49.716175 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.035770    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 22 02:20:49.716175 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.038201    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 22 02:20:49.716175 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.040875    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 22 02:20:49.716175 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.052169    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 22 02:20:49.716175 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.054740    2565 log.go:25] "Validated CRI v1 image API"
Jul 22 02:20:49.716175 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.055594    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 22 02:20:49.716175 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.060523    2565 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b322706c-2ff0-4aea-b3f8-f700fa74b1b8:/dev/nvme0n1p4 f2782a2c-0feb-4adf-a878-dcf3bc67ec14:/dev/nvme0n1p3]
Jul 22 02:20:49.716514 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.060538    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 22 02:20:49.716637 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.064562    2565 manager.go:216] Machine: {Timestamp:2026-07-22 02:20:48.06409498 +0000 UTC m=+0.378323210 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100000 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24c5f6ba05f24fafc12802d8ee581f SystemUUID:ec24c5f6-ba05-f24f-afc1-2802d8ee581f BootID:d0ad850e-ae2c-4e8a-b7df-7a03dac9cbeb Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8a:b4:69:0d:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8a:b4:69:0d:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:af:65:60:64:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 22 02:20:49.716637 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.064659    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 22 02:20:49.716637 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.064726    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 22 02:20:49.716637 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.067659    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 22 02:20:49.716637 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.067682    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-200.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 22 02:20:49.716637 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.068375    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 22 02:20:49.716637 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.068385    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 22 02:20:49.716637 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.068392    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.069132    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.070233    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.071255    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.072875    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.073012    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.073038    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.073047    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.073058    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.073068    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.073099    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.073473    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.073488    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.076153    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.077797    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.077824    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.078739    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.078752    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.078758    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.078763    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.078772    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.078777    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.078785    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.078790    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.078812    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.078817    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.078830    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.078838    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.078862    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.078867    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.086851    2565 server.go:1268] "Started kubelet"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.086909    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.086942    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.087032    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.087856    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g94jv"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:48.088100    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:48.088123    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-200.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.088136    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-200.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.088836    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 22 02:20:49.716974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.092221    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:48.092329    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:48.091345    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-200.ec2.internal.18c47b3b8f86b652  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-200.ec2.internal,UID:ip-10-0-140-200.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-200.ec2.internal,},FirstTimestamp:2026-07-22 02:20:48.086816338 +0000 UTC m=+0.401044568,LastTimestamp:2026-07-22 02:20:48.086816338 +0000 UTC m=+0.401044568,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-200.ec2.internal,}"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.095848    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g94jv"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.104852    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.104864    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:48.104998    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.105477    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.105605    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.105618    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.105686    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.105695    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.105840    2565 factory.go:55] Registering systemd factory
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.105853    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.106077    2565 factory.go:153] Registering CRI-O factory
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.106088    2565 factory.go:223] Registration of the crio container factory successfully
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.106127    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.106149    2565 factory.go:103] Registering Raw factory
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.106160    2565 manager.go:1196] Started watching for new ooms in manager
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.106450    2565 manager.go:349] Starting recovery of all containers
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.115409    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:48.118255    2565 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-200.ec2.internal\" not found" node="ip-10-0-140-200.ec2.internal"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.118916    2565 manager.go:354] Recovery completed
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.124595    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.126877    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.126907    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.126921    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientPID"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.127416    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.127430    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.127448    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.127450    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.128159    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.128182    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.128192    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientPID"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.128494    2565 policy_none.go:49] "None policy: Start"
Jul 22 02:20:49.718250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.128506    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 22 02:20:49.719554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.128516    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 22 02:20:49.719554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.132446    2565 policy_none.go:47] "Start"
Jul 22 02:20:49.719554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.180552    2565 manager.go:337] "Starting Device Plugin manager"
Jul 22 02:20:49.719554 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:48.180587    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 22 02:20:49.719554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.180600    2565 server.go:87] "Starting device plugin registration server"
Jul 22 02:20:49.719554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.180865    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 22 02:20:49.719554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.181019    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 22 02:20:49.719554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.181133    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 22 02:20:49.719554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.181226    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 22 02:20:49.719554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:48.181240    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 22 02:20:49.719554 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:49.647668    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 22 02:20:49.719554 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:49.647713    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 22 02:20:49.719554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.718233    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 22 02:20:49.719554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.719426    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 22 02:20:49.719554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.719464    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 22 02:20:49.719554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.719485    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 22 02:20:49.719554 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:49.719518    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.721602    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.747580    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.748395    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.748431    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.748442    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientPID"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.748464    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-200.ec2.internal"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.756968    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-200.ec2.internal"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:49.756985    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-200.ec2.internal\": node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:49.772303    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.820331    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal"]
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.820401    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.824191    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.824215    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.824227    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientPID"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.826487    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.826630    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.826660    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.827203    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.827229    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.827204    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.827261    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.827274    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientPID"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.827242    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientPID"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.829434    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.829460    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.830129    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.830155    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.830164    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeHasSufficientPID"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:49.845357    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-200.ec2.internal\" not found" node="ip-10-0-140-200.ec2.internal"
Jul 22 02:20:49.894773 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:49.852814    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-200.ec2.internal\" not found" node="ip-10-0-140-200.ec2.internal"
Jul 22 02:20:49.896386 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.872298    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e2f093a36821ff3dcaa0f49fa074849-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal\" (UID: \"9e2f093a36821ff3dcaa0f49fa074849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal"
Jul 22 02:20:49.896386 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.872318    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e2f093a36821ff3dcaa0f49fa074849-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal\" (UID: \"9e2f093a36821ff3dcaa0f49fa074849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal"
Jul 22 02:20:49.896386 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:49.872355    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 22 02:20:50.107639 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:49.972628    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 22 02:20:50.107639 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.972670    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e2f093a36821ff3dcaa0f49fa074849-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal\" (UID: \"9e2f093a36821ff3dcaa0f49fa074849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal"
Jul 22 02:20:50.107639 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.972691    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e2f093a36821ff3dcaa0f49fa074849-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal\" (UID: \"9e2f093a36821ff3dcaa0f49fa074849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal"
Jul 22 02:20:50.107639 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.972710    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0b30ae5bc27eeb039f1a3250327b5a6-config\") pod \"kube-apiserver-proxy-ip-10-0-140-200.ec2.internal\" (UID: \"a0b30ae5bc27eeb039f1a3250327b5a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal"
Jul 22 02:20:50.107639 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.972747    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e2f093a36821ff3dcaa0f49fa074849-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal\" (UID: \"9e2f093a36821ff3dcaa0f49fa074849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal"
Jul 22 02:20:50.107639 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:49.972757    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e2f093a36821ff3dcaa0f49fa074849-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal\" (UID: \"9e2f093a36821ff3dcaa0f49fa074849\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal"
Jul 22 02:20:50.107639 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:50.072767    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 22 02:20:50.107639 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:50.072786    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0b30ae5bc27eeb039f1a3250327b5a6-config\") pod \"kube-apiserver-proxy-ip-10-0-140-200.ec2.internal\" (UID: \"a0b30ae5bc27eeb039f1a3250327b5a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal"
Jul 22 02:20:50.107639 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:50.072811    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0b30ae5bc27eeb039f1a3250327b5a6-config\") pod \"kube-apiserver-proxy-ip-10-0-140-200.ec2.internal\" (UID: \"a0b30ae5bc27eeb039f1a3250327b5a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal"
Jul 22 02:20:50.280671 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:50.148983    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal"
Jul 22 02:20:50.280671 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:50.155246    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal"
Jul 22 02:20:50.280671 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:50.173759    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 22 02:20:50.280671 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:50.274110    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 22 02:20:50.480658 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:50.374771    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 22 02:20:50.480658 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:50.475449    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 22 02:20:50.678346 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:50.503631    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 22 02:20:50.678346 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:50.503768    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 22 02:20:50.678346 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:50.503797    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 22 02:20:50.678346 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:50.571683    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 02:15:49 +0000 UTC" deadline="2028-04-04 05:24:15.221735726 +0000 UTC"
Jul 22 02:20:50.678346 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:50.571710    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14931h3m24.650028819s"
Jul 22 02:20:50.678346 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:50.575486    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 22 02:20:50.678346 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:50.676373    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 22 02:20:50.876568 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:50.777278    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-200.ec2.internal\" not found"
Jul 22 02:20:50.876568 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:50.793979    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 02:20:50.876568 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:50.871382    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal"
Jul 22 02:20:50.882596 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:50.882573    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 22 02:20:50.884761 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:50.884737    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal"
Jul 22 02:20:50.894312 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:50.894294    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 22 02:20:50.982258 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:50.982236    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 02:20:50.985298 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:50.985275    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2f093a36821ff3dcaa0f49fa074849.slice/crio-fc805eceb572d05b3f55009774cba79e671ff90cc07cd2c0e9cf75318e87b322 WatchSource:0}: Error finding container fc805eceb572d05b3f55009774cba79e671ff90cc07cd2c0e9cf75318e87b322: Status 404 returned error can't find the container with id fc805eceb572d05b3f55009774cba79e671ff90cc07cd2c0e9cf75318e87b322
Jul 22 02:20:50.985709 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:50.985686    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b30ae5bc27eeb039f1a3250327b5a6.slice/crio-7b56b78338cba1de512f06869ee773b7a24394d17d0d59126afd0e3f4b8acfdf WatchSource:0}: Error finding container 7b56b78338cba1de512f06869ee773b7a24394d17d0d59126afd0e3f4b8acfdf: Status 404 returned error can't find the container with id 7b56b78338cba1de512f06869ee773b7a24394d17d0d59126afd0e3f4b8acfdf
Jul 22 02:20:50.996415 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:50.996399    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 02:20:51.540734 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.540695    2565 apiserver.go:52] "Watching apiserver"
Jul 22 02:20:51.550040 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.550015    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 22 02:20:51.550395 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.550372    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-4ggf4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7","openshift-cluster-node-tuning-operator/tuned-jkrrj","openshift-dns/node-resolver-zhtxk","openshift-image-registry/node-ca-v6sck","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal","openshift-multus/multus-additional-cni-plugins-pnw2f","openshift-multus/multus-kvk2x","kube-system/global-pull-secret-syncer-kzv8x","kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal","openshift-multus/network-metrics-daemon-p8jzv","openshift-network-diagnostics/network-check-target-r4hdg","openshift-network-operator/iptables-alerter-r7b6z","openshift-ovn-kubernetes/ovnkube-node-ssvw4"]
Jul 22 02:20:51.552972 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.552951    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.555872 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.555631    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 22 02:20:51.555872 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.555750    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 22 02:20:51.556023 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.555937    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 22 02:20:51.556023 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.555970    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 22 02:20:51.556237 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.556179    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 22 02:20:51.556237 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.556224    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r7b6z"
Jul 22 02:20:51.556380 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.556285    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kxwh5\""
Jul 22 02:20:51.559071 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.558645    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.559071 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.559034    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8zhn6\""
Jul 22 02:20:51.559071 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.559047    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 22 02:20:51.559298 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.559141    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 22 02:20:51.559298 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.559285    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 22 02:20:51.560795 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.560778    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zhtxk"
Jul 22 02:20:51.561402 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.561023    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sbz24\""
Jul 22 02:20:51.561402 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.561114    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 22 02:20:51.561402 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.561255    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 22 02:20:51.563470 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.563127    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 22 02:20:51.563470 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.563430    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6qlxn\""
Jul 22 02:20:51.563625 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.563487    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 22 02:20:51.563694 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.563679    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v6sck"
Jul 22 02:20:51.565944 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.565926    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 22 02:20:51.566669 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.566264    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 22 02:20:51.566669 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.566304    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 22 02:20:51.566669 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.566352    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-phhzt\""
Jul 22 02:20:51.566669 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.566286    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.568692 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.568677    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 22 02:20:51.568772 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.568709    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 22 02:20:51.568999 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.568981    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 22 02:20:51.569090 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.568999    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 22 02:20:51.569148 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.569095    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:20:51.569201 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.569173    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 22 02:20:51.569201 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:51.569178    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzv8x" podUID="faebb5f3-7db6-4e1c-88b0-a4530442e728"
Jul 22 02:20:51.569473 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.569422    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k7v8w\""
Jul 22 02:20:51.569550 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.569532    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 22 02:20:51.574689 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.572435    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4ggf4"
Jul 22 02:20:51.574689 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.573146    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 22 02:20:51.575270 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.575251    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 22 02:20:51.576208 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.575797    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 22 02:20:51.576208 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.576156    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mlvgv\""
Jul 22 02:20:51.576484 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.576465    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.578901 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.578882    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.579969 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579308    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-tmp\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.579969 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579342    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/665c3881-a42c-4856-a81a-d85782af2571-ovnkube-script-lib\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.579969 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579367    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdx8m\" (UniqueName: \"kubernetes.io/projected/665c3881-a42c-4856-a81a-d85782af2571-kube-api-access-wdx8m\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.579969 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/faebb5f3-7db6-4e1c-88b0-a4530442e728-dbus\") pod \"global-pull-secret-syncer-kzv8x\" (UID: \"faebb5f3-7db6-4e1c-88b0-a4530442e728\") " pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:20:51.579969 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579405    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cn58s\""
Jul 22 02:20:51.579969 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579412    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/665c3881-a42c-4856-a81a-d85782af2571-ovn-node-metrics-cert\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.579969 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579435    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-cni-bin\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.579969 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579458    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qbx5\" (UniqueName: \"kubernetes.io/projected/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-kube-api-access-6qbx5\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.579969 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579480    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-run-netns\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.579969 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579505    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-cni-netd\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.579969 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579526    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f5729d9-bc97-4082-8527-cc5473d03a02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.579969 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579548    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f5729d9-bc97-4082-8527-cc5473d03a02-registration-dir\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.579969 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579569    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f5729d9-bc97-4082-8527-cc5473d03a02-device-dir\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.579969 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579591    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f5729d9-bc97-4082-8527-cc5473d03a02-sys-fs\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.580731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579614    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5be685a-0b0f-4058-9272-2fb16817f84d-tmp-dir\") pod \"node-resolver-zhtxk\" (UID: \"f5be685a-0b0f-4058-9272-2fb16817f84d\") " pod="openshift-dns/node-resolver-zhtxk"
Jul 22 02:20:51.580731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579635    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1a137a5-3d05-45cf-8620-7537635e87d0-host\") pod \"node-ca-v6sck\" (UID: \"a1a137a5-3d05-45cf-8620-7537635e87d0\") " pod="openshift-image-registry/node-ca-v6sck"
Jul 22 02:20:51.580731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579658    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 22 02:20:51.580731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579675    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f5729d9-bc97-4082-8527-cc5473d03a02-socket-dir\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.580731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579699    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b9jk\" (UniqueName: \"kubernetes.io/projected/4f5729d9-bc97-4082-8527-cc5473d03a02-kube-api-access-4b9jk\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.580731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579725    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.580731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579749    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d9824be5-cf4c-49b5-9ad4-db70c783d24c-konnectivity-ca\") pod \"konnectivity-agent-4ggf4\" (UID: \"d9824be5-cf4c-49b5-9ad4-db70c783d24c\") " pod="kube-system/konnectivity-agent-4ggf4"
Jul 22 02:20:51.580731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579763    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 22 02:20:51.580731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579771    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-sys\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.580731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579794    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-etc-openvswitch\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.580731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579816    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-run-ovn\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.580731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579891    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-sysconfig\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.580731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579918    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-run-systemd\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.580731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.579970    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-node-log\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.580731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580009    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a07ecaa-d731-485d-a2d5-b28a59e7dee4-host-slash\") pod \"iptables-alerter-r7b6z\" (UID: \"9a07ecaa-d731-485d-a2d5-b28a59e7dee4\") " pod="openshift-network-operator/iptables-alerter-r7b6z"
Jul 22 02:20:51.581472 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580060    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-os-release\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.581472 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580098    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-kubernetes\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.581472 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580157    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-sysctl-d\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.581472 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580191    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-run\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.581472 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580234    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-run-ovn-kubernetes\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.581472 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580260    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/665c3881-a42c-4856-a81a-d85782af2571-env-overrides\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.581472 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.581472 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580348    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-systemd\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.581472 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580370    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgsn7\" (UniqueName: \"kubernetes.io/projected/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-kube-api-access-tgsn7\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.581472 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580413    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-var-lib-openvswitch\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.581472 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580438    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-cnibin\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.581472 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580472    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 22 02:20:51.581472 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580474    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9a07ecaa-d731-485d-a2d5-b28a59e7dee4-iptables-alerter-script\") pod \"iptables-alerter-r7b6z\" (UID: \"9a07ecaa-d731-485d-a2d5-b28a59e7dee4\") " pod="openshift-network-operator/iptables-alerter-r7b6z"
Jul 22 02:20:51.581472 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580556    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56sj8\" (UniqueName: \"kubernetes.io/projected/9a07ecaa-d731-485d-a2d5-b28a59e7dee4-kube-api-access-56sj8\") pod \"iptables-alerter-r7b6z\" (UID: \"9a07ecaa-d731-485d-a2d5-b28a59e7dee4\") " pod="openshift-network-operator/iptables-alerter-r7b6z"
Jul 22 02:20:51.582141 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580584    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-tuned\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.582141 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580607    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a1a137a5-3d05-45cf-8620-7537635e87d0-serviceca\") pod \"node-ca-v6sck\" (UID: \"a1a137a5-3d05-45cf-8620-7537635e87d0\") " pod="openshift-image-registry/node-ca-v6sck"
Jul 22 02:20:51.582141 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580634    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-system-cni-dir\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.582141 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580672    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.582141 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580705    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d9824be5-cf4c-49b5-9ad4-db70c783d24c-agent-certs\") pod \"konnectivity-agent-4ggf4\" (UID: \"d9824be5-cf4c-49b5-9ad4-db70c783d24c\") " pod="kube-system/konnectivity-agent-4ggf4"
Jul 22 02:20:51.582141 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580732    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-var-lib-kubelet\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.582141 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580755    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-kubelet\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.582141 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580778    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.582141 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580804    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-cni-binary-copy\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.582141 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-modprobe-d\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.582141 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580890    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-sysctl-conf\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.582141 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580912    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-host\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.582141 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580934    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8dcb\" (UniqueName: \"kubernetes.io/projected/a1a137a5-3d05-45cf-8620-7537635e87d0-kube-api-access-m8dcb\") pod \"node-ca-v6sck\" (UID: \"a1a137a5-3d05-45cf-8620-7537635e87d0\") " pod="openshift-image-registry/node-ca-v6sck"
Jul 22 02:20:51.582780 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580956    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-systemd-units\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.582780 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580978    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-run-openvswitch\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.582780 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.580999    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-log-socket\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.582780 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.581030    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f5729d9-bc97-4082-8527-cc5473d03a02-etc-selinux\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.582780 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.581060    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/faebb5f3-7db6-4e1c-88b0-a4530442e728-kubelet-config\") pod \"global-pull-secret-syncer-kzv8x\" (UID: \"faebb5f3-7db6-4e1c-88b0-a4530442e728\") " pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:20:51.582780 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.581085    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret\") pod \"global-pull-secret-syncer-kzv8x\" (UID: \"faebb5f3-7db6-4e1c-88b0-a4530442e728\") " pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:20:51.582780 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.581119    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-slash\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.582780 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.581140    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/665c3881-a42c-4856-a81a-d85782af2571-ovnkube-config\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.582780 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.581219    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f5be685a-0b0f-4058-9272-2fb16817f84d-hosts-file\") pod \"node-resolver-zhtxk\" (UID: \"f5be685a-0b0f-4058-9272-2fb16817f84d\") " pod="openshift-dns/node-resolver-zhtxk"
Jul 22 02:20:51.582780 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.581250    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wcdc\" (UniqueName: \"kubernetes.io/projected/f5be685a-0b0f-4058-9272-2fb16817f84d-kube-api-access-7wcdc\") pod \"node-resolver-zhtxk\" (UID: \"f5be685a-0b0f-4058-9272-2fb16817f84d\") " pod="openshift-dns/node-resolver-zhtxk"
Jul 22 02:20:51.582780 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.581274    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-lib-modules\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.582780 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.581565    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 22 02:20:51.582780 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.581586    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-82xzn\""
Jul 22 02:20:51.582780 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.581818    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:20:51.582780 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:51.581907    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8jzv" podUID="ba53539c-579d-43a1-a6d0-d9020f0d18f3"
Jul 22 02:20:51.583753 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.583729    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:20:51.583835 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:51.583799    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4hdg" podUID="0036981e-4941-42f3-91d1-d1c98ffa1f5e"
Jul 22 02:20:51.584242 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.584198    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 22 02:20:51.605794 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.605773    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gn8zf"
Jul 22 02:20:51.614967 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.614909    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gn8zf"
Jul 22 02:20:51.672522 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.672499    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 22 02:20:51.682410 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.682384    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-system-cni-dir\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.682996 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.682423    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.682996 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.682468    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs\") pod \"network-metrics-daemon-p8jzv\" (UID: \"ba53539c-579d-43a1-a6d0-d9020f0d18f3\") " pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:20:51.682996 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.682493    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-os-release\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.682996 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.682519    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d9824be5-cf4c-49b5-9ad4-db70c783d24c-agent-certs\") pod \"konnectivity-agent-4ggf4\" (UID: \"d9824be5-cf4c-49b5-9ad4-db70c783d24c\") " pod="kube-system/konnectivity-agent-4ggf4"
Jul 22 02:20:51.682996 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.682565    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-var-lib-kubelet\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.682996 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.682593    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-kubelet\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.682996 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.682618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.682996 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.682652    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-cni-binary-copy\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.682996 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.682676    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-host-run-netns\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.682996 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.682701    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-host-var-lib-cni-bin\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.682996 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.682724    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-modprobe-d\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.682996 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.682749    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-sysctl-conf\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.682996 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.682771    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-host\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.682996 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.682864    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8dcb\" (UniqueName: \"kubernetes.io/projected/a1a137a5-3d05-45cf-8620-7537635e87d0-kube-api-access-m8dcb\") pod \"node-ca-v6sck\" (UID: \"a1a137a5-3d05-45cf-8620-7537635e87d0\") " pod="openshift-image-registry/node-ca-v6sck"
Jul 22 02:20:51.683658 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.682890    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-systemd-units\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.683658 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.682919    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-run-openvswitch\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.683658 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.682894    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-var-lib-kubelet\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.683658 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.682940    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-log-socket\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.683658 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.682986    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-system-cni-dir\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.683658 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683001    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-log-socket\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.683658 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683016    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f5729d9-bc97-4082-8527-cc5473d03a02-etc-selinux\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.683658 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683059    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vslzw\" (UniqueName: \"kubernetes.io/projected/ba53539c-579d-43a1-a6d0-d9020f0d18f3-kube-api-access-vslzw\") pod \"network-metrics-daemon-p8jzv\" (UID: \"ba53539c-579d-43a1-a6d0-d9020f0d18f3\") " pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:20:51.683658 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683086    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-host-run-k8s-cni-cncf-io\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.683658 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683109    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-host-run-multus-certs\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.683658 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683132    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hchkl\" (UniqueName: \"kubernetes.io/projected/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-kube-api-access-hchkl\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.683658 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683137    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-kubelet\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.683658 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683158    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/faebb5f3-7db6-4e1c-88b0-a4530442e728-kubelet-config\") pod \"global-pull-secret-syncer-kzv8x\" (UID: \"faebb5f3-7db6-4e1c-88b0-a4530442e728\") " pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:20:51.683658 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683181    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.684329 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683185    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret\") pod \"global-pull-secret-syncer-kzv8x\" (UID: \"faebb5f3-7db6-4e1c-88b0-a4530442e728\") " pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:20:51.684329 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683216    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-slash\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.684329 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683241    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/665c3881-a42c-4856-a81a-d85782af2571-ovnkube-config\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.684329 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:51.683264    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 02:20:51.684329 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683266    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f5be685a-0b0f-4058-9272-2fb16817f84d-hosts-file\") pod \"node-resolver-zhtxk\" (UID: \"f5be685a-0b0f-4058-9272-2fb16817f84d\") " pod="openshift-dns/node-resolver-zhtxk"
Jul 22 02:20:51.684329 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683313    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wcdc\" (UniqueName: \"kubernetes.io/projected/f5be685a-0b0f-4058-9272-2fb16817f84d-kube-api-access-7wcdc\") pod \"node-resolver-zhtxk\" (UID: \"f5be685a-0b0f-4058-9272-2fb16817f84d\") " pod="openshift-dns/node-resolver-zhtxk"
Jul 22 02:20:51.684329 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683340    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-multus-cni-dir\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.684329 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683365    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-lib-modules\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.684329 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683388    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-tmp\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.684329 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/665c3881-a42c-4856-a81a-d85782af2571-ovnkube-script-lib\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.684329 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683437    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdx8m\" (UniqueName: \"kubernetes.io/projected/665c3881-a42c-4856-a81a-d85782af2571-kube-api-access-wdx8m\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.684329 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683462    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdmgc\" (UniqueName: \"kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc\") pod \"network-check-target-r4hdg\" (UID: \"0036981e-4941-42f3-91d1-d1c98ffa1f5e\") " pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:20:51.684329 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683486    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/faebb5f3-7db6-4e1c-88b0-a4530442e728-dbus\") pod \"global-pull-secret-syncer-kzv8x\" (UID: \"faebb5f3-7db6-4e1c-88b0-a4530442e728\") " pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:20:51.684329 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/665c3881-a42c-4856-a81a-d85782af2571-ovn-node-metrics-cert\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.684329 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-multus-socket-dir-parent\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.685056 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683563    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-host-var-lib-kubelet\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.685056 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683571    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.685056 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-cni-bin\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.685056 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683620    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qbx5\" (UniqueName: \"kubernetes.io/projected/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-kube-api-access-6qbx5\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.685056 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683643    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-run-netns\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.685056 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683666    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-cni-netd\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.685056 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683708    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f5729d9-bc97-4082-8527-cc5473d03a02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.685056 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683733    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-hostroot\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.685056 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683755    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-multus-daemon-config\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.685056 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683780    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f5729d9-bc97-4082-8527-cc5473d03a02-registration-dir\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.685056 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683827    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f5729d9-bc97-4082-8527-cc5473d03a02-device-dir\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.685056 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683871    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f5729d9-bc97-4082-8527-cc5473d03a02-sys-fs\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.685056 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683897    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5be685a-0b0f-4058-9272-2fb16817f84d-tmp-dir\") pod \"node-resolver-zhtxk\" (UID: \"f5be685a-0b0f-4058-9272-2fb16817f84d\") " pod="openshift-dns/node-resolver-zhtxk"
Jul 22 02:20:51.685056 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683890    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 22 02:20:51.685699 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683962    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-system-cni-dir\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.685699 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.683989    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-host-var-lib-cni-multus\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.685699 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684014    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1a137a5-3d05-45cf-8620-7537635e87d0-host\") pod \"node-ca-v6sck\" (UID: \"a1a137a5-3d05-45cf-8620-7537635e87d0\") " pod="openshift-image-registry/node-ca-v6sck"
Jul 22 02:20:51.685699 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684038    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f5729d9-bc97-4082-8527-cc5473d03a02-socket-dir\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.685699 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684064    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b9jk\" (UniqueName: \"kubernetes.io/projected/4f5729d9-bc97-4082-8527-cc5473d03a02-kube-api-access-4b9jk\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.685699 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684100    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.685699 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684129    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d9824be5-cf4c-49b5-9ad4-db70c783d24c-konnectivity-ca\") pod \"konnectivity-agent-4ggf4\" (UID: \"d9824be5-cf4c-49b5-9ad4-db70c783d24c\") " pod="kube-system/konnectivity-agent-4ggf4"
Jul 22 02:20:51.685699 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684155    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-sys\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.685699 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684181    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-etc-openvswitch\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.685699 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684207    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-run-ovn\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.685699 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684257    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-sysconfig\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.685699 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684281    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-run-systemd\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.685699 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684305    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-node-log\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.685699 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684334    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a07ecaa-d731-485d-a2d5-b28a59e7dee4-host-slash\") pod \"iptables-alerter-r7b6z\" (UID: \"9a07ecaa-d731-485d-a2d5-b28a59e7dee4\") " pod="openshift-network-operator/iptables-alerter-r7b6z"
Jul 22 02:20:51.686302 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684360    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-os-release\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.686302 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684386    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-kubernetes\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.686302 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684411    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-sysctl-d\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.686302 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684435    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-run\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.686302 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684460    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-run-ovn-kubernetes\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.686302 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/665c3881-a42c-4856-a81a-d85782af2571-env-overrides\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.686302 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684511    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.686302 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684554    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-cnibin\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.686302 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684581    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-etc-kubernetes\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.686302 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684607    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-systemd\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.686302 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684631    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgsn7\" (UniqueName: \"kubernetes.io/projected/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-kube-api-access-tgsn7\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.686302 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684659    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-var-lib-openvswitch\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.686302 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684714    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-var-lib-openvswitch\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.686302 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684761    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f5729d9-bc97-4082-8527-cc5473d03a02-etc-selinux\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.686990 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684815    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-cnibin\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.686990 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684841    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-cni-binary-copy\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.686990 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684878    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-multus-conf-dir\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.686990 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684904    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9a07ecaa-d731-485d-a2d5-b28a59e7dee4-iptables-alerter-script\") pod \"iptables-alerter-r7b6z\" (UID: \"9a07ecaa-d731-485d-a2d5-b28a59e7dee4\") " pod="openshift-network-operator/iptables-alerter-r7b6z"
Jul 22 02:20:51.686990 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684929    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56sj8\" (UniqueName: \"kubernetes.io/projected/9a07ecaa-d731-485d-a2d5-b28a59e7dee4-kube-api-access-56sj8\") pod \"iptables-alerter-r7b6z\" (UID: \"9a07ecaa-d731-485d-a2d5-b28a59e7dee4\") " pod="openshift-network-operator/iptables-alerter-r7b6z"
Jul 22 02:20:51.686990 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-tuned\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.686990 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.684979    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a1a137a5-3d05-45cf-8620-7537635e87d0-serviceca\") pod \"node-ca-v6sck\" (UID: \"a1a137a5-3d05-45cf-8620-7537635e87d0\") " pod="openshift-image-registry/node-ca-v6sck"
Jul 22 02:20:51.686990 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.685267    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-cni-binary-copy\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.686990 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.685403    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a1a137a5-3d05-45cf-8620-7537635e87d0-serviceca\") pod \"node-ca-v6sck\" (UID: \"a1a137a5-3d05-45cf-8620-7537635e87d0\") " pod="openshift-image-registry/node-ca-v6sck"
Jul 22 02:20:51.686990 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.685447    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-modprobe-d\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.686990 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.685494    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/faebb5f3-7db6-4e1c-88b0-a4530442e728-kubelet-config\") pod \"global-pull-secret-syncer-kzv8x\" (UID: \"faebb5f3-7db6-4e1c-88b0-a4530442e728\") " pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:20:51.686990 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.685546    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-sysctl-conf\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.686990 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.685549    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-cnibin\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.686990 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.685600    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-host\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.686990 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.685662    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-run-systemd\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.687752 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.685692    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f5729d9-bc97-4082-8527-cc5473d03a02-sys-fs\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.687752 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.685747    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-systemd-units\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.687752 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.685792    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-run-openvswitch\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.687752 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.685829    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-slash\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.687752 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.685881    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-node-log\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.687752 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.685919    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a07ecaa-d731-485d-a2d5-b28a59e7dee4-host-slash\") pod \"iptables-alerter-r7b6z\" (UID: \"9a07ecaa-d731-485d-a2d5-b28a59e7dee4\") " pod="openshift-network-operator/iptables-alerter-r7b6z"
Jul 22 02:20:51.687752 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.685935    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-os-release\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.687752 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.685943    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5be685a-0b0f-4058-9272-2fb16817f84d-tmp-dir\") pod \"node-resolver-zhtxk\" (UID: \"f5be685a-0b0f-4058-9272-2fb16817f84d\") " pod="openshift-dns/node-resolver-zhtxk"
Jul 22 02:20:51.687752 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.685994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-kubernetes\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.687752 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.686081    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-sysctl-d\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.687752 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.686081    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f5729d9-bc97-4082-8527-cc5473d03a02-socket-dir\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.687752 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.686081    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9a07ecaa-d731-485d-a2d5-b28a59e7dee4-iptables-alerter-script\") pod \"iptables-alerter-r7b6z\" (UID: \"9a07ecaa-d731-485d-a2d5-b28a59e7dee4\") " pod="openshift-network-operator/iptables-alerter-r7b6z"
Jul 22 02:20:51.687752 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.686113    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1a137a5-3d05-45cf-8620-7537635e87d0-host\") pod \"node-ca-v6sck\" (UID: \"a1a137a5-3d05-45cf-8620-7537635e87d0\") " pod="openshift-image-registry/node-ca-v6sck"
Jul 22 02:20:51.687752 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.686195    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-run\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.687752 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.686234    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-run-ovn-kubernetes\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.688487 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.686573    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/665c3881-a42c-4856-a81a-d85782af2571-env-overrides\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.688487 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.686920    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.688487 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.687025    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.688487 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.687139    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-systemd\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.688487 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.687838    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d9824be5-cf4c-49b5-9ad4-db70c783d24c-konnectivity-ca\") pod \"konnectivity-agent-4ggf4\" (UID: \"d9824be5-cf4c-49b5-9ad4-db70c783d24c\") " pod="kube-system/konnectivity-agent-4ggf4"
Jul 22 02:20:51.688487 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.688039    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/665c3881-a42c-4856-a81a-d85782af2571-ovnkube-config\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.688487 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:51.688238    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret podName:faebb5f3-7db6-4e1c-88b0-a4530442e728 nodeName:}" failed. No retries permitted until 2026-07-22 02:20:52.188212885 +0000 UTC m=+3.036549445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret") pod "global-pull-secret-syncer-kzv8x" (UID: "faebb5f3-7db6-4e1c-88b0-a4530442e728") : object "kube-system"/"original-pull-secret" not registered
Jul 22 02:20:51.688487 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.688466    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-cni-bin\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.688806 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.688512    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d9824be5-cf4c-49b5-9ad4-db70c783d24c-agent-certs\") pod \"konnectivity-agent-4ggf4\" (UID: \"d9824be5-cf4c-49b5-9ad4-db70c783d24c\") " pod="kube-system/konnectivity-agent-4ggf4"
Jul 22 02:20:51.688806 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.688648    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-run-netns\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.688806 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.688727    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-run-ovn\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.689017 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.688825    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f5be685a-0b0f-4058-9272-2fb16817f84d-hosts-file\") pod \"node-resolver-zhtxk\" (UID: \"f5be685a-0b0f-4058-9272-2fb16817f84d\") " pod="openshift-dns/node-resolver-zhtxk"
Jul 22 02:20:51.689017 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.688826    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-etc-openvswitch\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.689017 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.688957    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-lib-modules\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.689017 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.688972    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f5729d9-bc97-4082-8527-cc5473d03a02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.689201 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.689096    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f5729d9-bc97-4082-8527-cc5473d03a02-registration-dir\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.689201 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.689111    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/faebb5f3-7db6-4e1c-88b0-a4530442e728-dbus\") pod \"global-pull-secret-syncer-kzv8x\" (UID: \"faebb5f3-7db6-4e1c-88b0-a4530442e728\") " pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:20:51.689201 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.689171    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/665c3881-a42c-4856-a81a-d85782af2571-host-cni-netd\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.689201 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.689183    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f5729d9-bc97-4082-8527-cc5473d03a02-device-dir\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.689566 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.689541    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-tuned\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.692414 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.690942    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-sys\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.692414 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.691007    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-etc-sysconfig\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.692414 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.691269    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/665c3881-a42c-4856-a81a-d85782af2571-ovnkube-script-lib\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.694050 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.694000    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/665c3881-a42c-4856-a81a-d85782af2571-ovn-node-metrics-cert\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.694518 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.694390    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-tmp\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.695383 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.695361    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8dcb\" (UniqueName: \"kubernetes.io/projected/a1a137a5-3d05-45cf-8620-7537635e87d0-kube-api-access-m8dcb\") pod \"node-ca-v6sck\" (UID: \"a1a137a5-3d05-45cf-8620-7537635e87d0\") " pod="openshift-image-registry/node-ca-v6sck"
Jul 22 02:20:51.697036 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.697013    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgsn7\" (UniqueName: \"kubernetes.io/projected/41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b-kube-api-access-tgsn7\") pod \"tuned-jkrrj\" (UID: \"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b\") " pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.697525 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.697462    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b9jk\" (UniqueName: \"kubernetes.io/projected/4f5729d9-bc97-4082-8527-cc5473d03a02-kube-api-access-4b9jk\") pod \"aws-ebs-csi-driver-node-lmph7\" (UID: \"4f5729d9-bc97-4082-8527-cc5473d03a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.697979 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.697923    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qbx5\" (UniqueName: \"kubernetes.io/projected/32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b-kube-api-access-6qbx5\") pod \"multus-additional-cni-plugins-pnw2f\" (UID: \"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b\") " pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.699473 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.699431    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56sj8\" (UniqueName: \"kubernetes.io/projected/9a07ecaa-d731-485d-a2d5-b28a59e7dee4-kube-api-access-56sj8\") pod \"iptables-alerter-r7b6z\" (UID: \"9a07ecaa-d731-485d-a2d5-b28a59e7dee4\") " pod="openshift-network-operator/iptables-alerter-r7b6z"
Jul 22 02:20:51.700116 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.700072    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wcdc\" (UniqueName: \"kubernetes.io/projected/f5be685a-0b0f-4058-9272-2fb16817f84d-kube-api-access-7wcdc\") pod \"node-resolver-zhtxk\" (UID: \"f5be685a-0b0f-4058-9272-2fb16817f84d\") " pod="openshift-dns/node-resolver-zhtxk"
Jul 22 02:20:51.701025 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.700988    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdx8m\" (UniqueName: \"kubernetes.io/projected/665c3881-a42c-4856-a81a-d85782af2571-kube-api-access-wdx8m\") pod \"ovnkube-node-ssvw4\" (UID: \"665c3881-a42c-4856-a81a-d85782af2571\") " pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.723871 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.723811    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal" event={"ID":"a0b30ae5bc27eeb039f1a3250327b5a6","Type":"ContainerStarted","Data":"7b56b78338cba1de512f06869ee773b7a24394d17d0d59126afd0e3f4b8acfdf"}
Jul 22 02:20:51.725998 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.725973    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal" event={"ID":"9e2f093a36821ff3dcaa0f49fa074849","Type":"ContainerStarted","Data":"fc805eceb572d05b3f55009774cba79e671ff90cc07cd2c0e9cf75318e87b322"}
Jul 22 02:20:51.736509 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.736488    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 02:20:51.785569 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.785542    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-cnibin\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.785704 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.785575    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-etc-kubernetes\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.785704 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.785597    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-cni-binary-copy\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.785704 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.785619    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-multus-conf-dir\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.785704 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.785646    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs\") pod \"network-metrics-daemon-p8jzv\" (UID: \"ba53539c-579d-43a1-a6d0-d9020f0d18f3\") " pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:20:51.785704 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.785667    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-os-release\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.785704 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.785670    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-cnibin\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.785704 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.785694    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-host-run-netns\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.786066 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.785718    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-host-var-lib-cni-bin\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.786066 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.785721    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-multus-conf-dir\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.786066 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.785760    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-etc-kubernetes\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.786066 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:51.785835    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:20:51.786066 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:51.785912    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs podName:ba53539c-579d-43a1-a6d0-d9020f0d18f3 nodeName:}" failed. No retries permitted until 2026-07-22 02:20:52.285893899 +0000 UTC m=+3.134230467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs") pod "network-metrics-daemon-p8jzv" (UID: "ba53539c-579d-43a1-a6d0-d9020f0d18f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:20:51.786066 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.785995    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-os-release\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.786066 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786032    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-host-run-netns\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.786480 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786073    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-host-var-lib-cni-bin\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.786480 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786121    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vslzw\" (UniqueName: \"kubernetes.io/projected/ba53539c-579d-43a1-a6d0-d9020f0d18f3-kube-api-access-vslzw\") pod \"network-metrics-daemon-p8jzv\" (UID: \"ba53539c-579d-43a1-a6d0-d9020f0d18f3\") " pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:20:51.786480 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786342    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-cni-binary-copy\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.786480 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786382    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-host-run-k8s-cni-cncf-io\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.786480 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786428    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-host-run-k8s-cni-cncf-io\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.786480 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786455    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-host-run-multus-certs\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.786771 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786494    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-host-run-multus-certs\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.786771 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786499    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hchkl\" (UniqueName: \"kubernetes.io/projected/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-kube-api-access-hchkl\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.786771 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786542    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-multus-cni-dir\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.786771 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786569    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmgc\" (UniqueName: \"kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc\") pod \"network-check-target-r4hdg\" (UID: \"0036981e-4941-42f3-91d1-d1c98ffa1f5e\") " pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:20:51.786771 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786685    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-multus-socket-dir-parent\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.786771 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786713    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-host-var-lib-kubelet\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.786771 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786741    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-hostroot\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.786771 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786767    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-multus-daemon-config\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.787173 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786795    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-system-cni-dir\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.787173 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786799    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-host-var-lib-kubelet\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.787173 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786828    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-host-var-lib-cni-multus\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.787173 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786866    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-system-cni-dir\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.787173 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786919    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-multus-cni-dir\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.787173 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786919    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-multus-socket-dir-parent\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.787173 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786948    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-host-var-lib-cni-multus\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.787173 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.786959    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-hostroot\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.787947 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.787909    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-multus-daemon-config\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.792618 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:51.792295    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 02:20:51.792618 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:51.792319    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 02:20:51.792618 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:51.792332    2565 projected.go:196] Error preparing data for projected volume kube-api-access-sdmgc for pod openshift-network-diagnostics/network-check-target-r4hdg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:20:51.792618 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:51.792404    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc podName:0036981e-4941-42f3-91d1-d1c98ffa1f5e nodeName:}" failed. No retries permitted until 2026-07-22 02:20:52.292388515 +0000 UTC m=+3.140725089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sdmgc" (UniqueName: "kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc") pod "network-check-target-r4hdg" (UID: "0036981e-4941-42f3-91d1-d1c98ffa1f5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:20:51.796154 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.796130    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hchkl\" (UniqueName: \"kubernetes.io/projected/e736b988-93b9-40f9-9ab3-e3ee1a72f5a8-kube-api-access-hchkl\") pod \"multus-kvk2x\" (UID: \"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8\") " pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:51.796749 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.796597    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vslzw\" (UniqueName: \"kubernetes.io/projected/ba53539c-579d-43a1-a6d0-d9020f0d18f3-kube-api-access-vslzw\") pod \"network-metrics-daemon-p8jzv\" (UID: \"ba53539c-579d-43a1-a6d0-d9020f0d18f3\") " pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:20:51.867009 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.866974    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pnw2f"
Jul 22 02:20:51.878618 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.878588    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r7b6z"
Jul 22 02:20:51.887461 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.887438    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jkrrj"
Jul 22 02:20:51.893628 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.893608    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zhtxk"
Jul 22 02:20:51.901375 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.901348    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v6sck"
Jul 22 02:20:51.911119 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.911098    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:20:51.918223 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.918203    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4ggf4"
Jul 22 02:20:51.918519 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.918494    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 02:20:51.926882 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.926862    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7"
Jul 22 02:20:51.933609 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:51.933585    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kvk2x"
Jul 22 02:20:52.189820 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:52.189740    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret\") pod \"global-pull-secret-syncer-kzv8x\" (UID: \"faebb5f3-7db6-4e1c-88b0-a4530442e728\") " pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:20:52.189968 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:52.189887    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 02:20:52.189968 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:52.189951    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret podName:faebb5f3-7db6-4e1c-88b0-a4530442e728 nodeName:}" failed. No retries permitted until 2026-07-22 02:20:53.189933817 +0000 UTC m=+4.038270385 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret") pod "global-pull-secret-syncer-kzv8x" (UID: "faebb5f3-7db6-4e1c-88b0-a4530442e728") : object "kube-system"/"original-pull-secret" not registered
Jul 22 02:20:52.290079 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:52.290043    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs\") pod \"network-metrics-daemon-p8jzv\" (UID: \"ba53539c-579d-43a1-a6d0-d9020f0d18f3\") " pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:20:52.290248 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:52.290209    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:20:52.290327 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:52.290272    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs podName:ba53539c-579d-43a1-a6d0-d9020f0d18f3 nodeName:}" failed. No retries permitted until 2026-07-22 02:20:53.290256257 +0000 UTC m=+4.138592808 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs") pod "network-metrics-daemon-p8jzv" (UID: "ba53539c-579d-43a1-a6d0-d9020f0d18f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:20:52.391182 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:52.391143    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmgc\" (UniqueName: \"kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc\") pod \"network-check-target-r4hdg\" (UID: \"0036981e-4941-42f3-91d1-d1c98ffa1f5e\") " pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:20:52.391359 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:52.391306    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 02:20:52.391359 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:52.391329    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 02:20:52.391359 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:52.391340    2565 projected.go:196] Error preparing data for projected volume kube-api-access-sdmgc for pod openshift-network-diagnostics/network-check-target-r4hdg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:20:52.391488 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:52.391394    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc podName:0036981e-4941-42f3-91d1-d1c98ffa1f5e nodeName:}" failed. No retries permitted until 2026-07-22 02:20:53.391377688 +0000 UTC m=+4.239714249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sdmgc" (UniqueName: "kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc") pod "network-check-target-r4hdg" (UID: "0036981e-4941-42f3-91d1-d1c98ffa1f5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:20:52.618410 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:52.618335    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 02:15:51 +0000 UTC" deadline="2028-01-27 06:50:01.430396591 +0000 UTC"
Jul 22 02:20:52.618410 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:52.618368    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13300h29m8.812032708s"
Jul 22 02:20:52.719954 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:52.719919    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:20:52.720447 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:52.719924    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:20:52.720447 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:52.720041    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4hdg" podUID="0036981e-4941-42f3-91d1-d1c98ffa1f5e"
Jul 22 02:20:52.720447 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:52.720157    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8jzv" podUID="ba53539c-579d-43a1-a6d0-d9020f0d18f3"
Jul 22 02:20:52.880954 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:52.880914    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32559c29_8fd8_4c7b_8bdc_aa9dc7607f3b.slice/crio-2f6ab52ee4867d2448ede1b6641e7c0e5bc1cdf76bb551b9719ed1f00ca39757 WatchSource:0}: Error finding container 2f6ab52ee4867d2448ede1b6641e7c0e5bc1cdf76bb551b9719ed1f00ca39757: Status 404 returned error can't find the container with id 2f6ab52ee4867d2448ede1b6641e7c0e5bc1cdf76bb551b9719ed1f00ca39757
Jul 22 02:20:52.899101 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:52.899074    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a07ecaa_d731_485d_a2d5_b28a59e7dee4.slice/crio-79c1c52a8dced3d9bac24e20b976cc4a0b25682a5ccb99d78a01c9a355f689fa WatchSource:0}: Error finding container 79c1c52a8dced3d9bac24e20b976cc4a0b25682a5ccb99d78a01c9a355f689fa: Status 404 returned error can't find the container with id 79c1c52a8dced3d9bac24e20b976cc4a0b25682a5ccb99d78a01c9a355f689fa
Jul 22 02:20:52.900752 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:52.900729    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9824be5_cf4c_49b5_9ad4_db70c783d24c.slice/crio-50afaa83081525bead5a1a4bfe2b4add169cd4c8245eebc31fb6fcc924983a98 WatchSource:0}: Error finding container 50afaa83081525bead5a1a4bfe2b4add169cd4c8245eebc31fb6fcc924983a98: Status 404 returned error can't find the container with id 50afaa83081525bead5a1a4bfe2b4add169cd4c8245eebc31fb6fcc924983a98
Jul 22 02:20:52.901398 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:52.901379    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a137a5_3d05_45cf_8620_7537635e87d0.slice/crio-bdd39590930465ff0d0753106699e3d1b6b65041aff6517f1a9a6040d3d153a8 WatchSource:0}: Error finding container bdd39590930465ff0d0753106699e3d1b6b65041aff6517f1a9a6040d3d153a8: Status 404 returned error can't find the container with id bdd39590930465ff0d0753106699e3d1b6b65041aff6517f1a9a6040d3d153a8
Jul 22 02:20:52.902631 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:52.902584    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665c3881_a42c_4856_a81a_d85782af2571.slice/crio-86ca84e0bced468c19afa8a7e2df935e76695f5fe5eba41ce08a4cf48bae5885 WatchSource:0}: Error finding container 86ca84e0bced468c19afa8a7e2df935e76695f5fe5eba41ce08a4cf48bae5885: Status 404 returned error can't find the container with id 86ca84e0bced468c19afa8a7e2df935e76695f5fe5eba41ce08a4cf48bae5885
Jul 22 02:20:52.903826 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:52.903426    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5729d9_bc97_4082_8527_cc5473d03a02.slice/crio-343c3ae8e2083e57d5f79d2c54f626981b7519a4045d4b2d499fdb1feeee5a31 WatchSource:0}: Error finding container 343c3ae8e2083e57d5f79d2c54f626981b7519a4045d4b2d499fdb1feeee5a31: Status 404 returned error can't find the container with id 343c3ae8e2083e57d5f79d2c54f626981b7519a4045d4b2d499fdb1feeee5a31
Jul 22 02:20:52.905090 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:20:52.905031    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41818c5b_aa1c_42c2_ab5d_9d37d9a8e23b.slice/crio-647f972a584834952b9622a3ea86e0273647bab63c07e9a1ab914a38489d4dd9 WatchSource:0}: Error finding container 647f972a584834952b9622a3ea86e0273647bab63c07e9a1ab914a38489d4dd9: Status 404 returned error can't find the container with id 647f972a584834952b9622a3ea86e0273647bab63c07e9a1ab914a38489d4dd9
Jul 22 02:20:53.196787 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:53.196702    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret\") pod \"global-pull-secret-syncer-kzv8x\" (UID: \"faebb5f3-7db6-4e1c-88b0-a4530442e728\") " pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:20:53.196907 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:53.196812    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 02:20:53.196907 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:53.196880    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret podName:faebb5f3-7db6-4e1c-88b0-a4530442e728 nodeName:}" failed. No retries permitted until 2026-07-22 02:20:55.196865288 +0000 UTC m=+6.045201856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret") pod "global-pull-secret-syncer-kzv8x" (UID: "faebb5f3-7db6-4e1c-88b0-a4530442e728") : object "kube-system"/"original-pull-secret" not registered
Jul 22 02:20:53.298019 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:53.297979    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs\") pod \"network-metrics-daemon-p8jzv\" (UID: \"ba53539c-579d-43a1-a6d0-d9020f0d18f3\") " pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:20:53.298186 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:53.298156    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:20:53.298256 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:53.298246    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs podName:ba53539c-579d-43a1-a6d0-d9020f0d18f3 nodeName:}" failed. No retries permitted until 2026-07-22 02:20:55.298225502 +0000 UTC m=+6.146562066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs") pod "network-metrics-daemon-p8jzv" (UID: "ba53539c-579d-43a1-a6d0-d9020f0d18f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:20:53.399208 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:53.399173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmgc\" (UniqueName: \"kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc\") pod \"network-check-target-r4hdg\" (UID: \"0036981e-4941-42f3-91d1-d1c98ffa1f5e\") " pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:20:53.399375 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:53.399352    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 02:20:53.399375 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:53.399369    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 02:20:53.399479 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:53.399382    2565 projected.go:196] Error preparing data for projected volume kube-api-access-sdmgc for pod openshift-network-diagnostics/network-check-target-r4hdg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:20:53.399479 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:53.399437    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc podName:0036981e-4941-42f3-91d1-d1c98ffa1f5e nodeName:}" failed. No retries permitted until 2026-07-22 02:20:55.399418621 +0000 UTC m=+6.247755187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sdmgc" (UniqueName: "kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc") pod "network-check-target-r4hdg" (UID: "0036981e-4941-42f3-91d1-d1c98ffa1f5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:20:53.619290 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:53.619183    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 02:15:51 +0000 UTC" deadline="2028-01-07 18:06:26.779201258 +0000 UTC"
Jul 22 02:20:53.619290 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:53.619246    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12831h45m33.159959092s"
Jul 22 02:20:53.721065 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:53.721031    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:20:53.721586 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:53.721168    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzv8x" podUID="faebb5f3-7db6-4e1c-88b0-a4530442e728"
Jul 22 02:20:53.738735 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:53.738700    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kvk2x" event={"ID":"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8","Type":"ContainerStarted","Data":"3645073ecb796ad6a3e5b0227aad27b903bfb3279b8d176476b430689af18d8b"}
Jul 22 02:20:53.745935 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:53.745896    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v6sck" event={"ID":"a1a137a5-3d05-45cf-8620-7537635e87d0","Type":"ContainerStarted","Data":"bdd39590930465ff0d0753106699e3d1b6b65041aff6517f1a9a6040d3d153a8"}
Jul 22 02:20:53.758510 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:53.758481    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4ggf4" event={"ID":"d9824be5-cf4c-49b5-9ad4-db70c783d24c","Type":"ContainerStarted","Data":"50afaa83081525bead5a1a4bfe2b4add169cd4c8245eebc31fb6fcc924983a98"}
Jul 22 02:20:53.775817 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:53.775729    2565 generic.go:356] "Generic (PLEG): container finished" podID="9e2f093a36821ff3dcaa0f49fa074849" containerID="cbf52236f6abf47c74933dbf4687adc05227c87bc8b98a6539a4fc91c322d86f" exitCode=0
Jul 22 02:20:53.775958 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:53.775815    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal" event={"ID":"9e2f093a36821ff3dcaa0f49fa074849","Type":"ContainerDied","Data":"cbf52236f6abf47c74933dbf4687adc05227c87bc8b98a6539a4fc91c322d86f"}
Jul 22 02:20:53.783982 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:53.783955    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zhtxk" event={"ID":"f5be685a-0b0f-4058-9272-2fb16817f84d","Type":"ContainerStarted","Data":"2561c73a51ff2b41f68964a6903103c54b14405319fcd8ff34c7043829f75b30"}
Jul 22 02:20:53.789921 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:53.789895    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jkrrj" event={"ID":"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b","Type":"ContainerStarted","Data":"647f972a584834952b9622a3ea86e0273647bab63c07e9a1ab914a38489d4dd9"}
Jul 22 02:20:53.792827 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:53.792761    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7" event={"ID":"4f5729d9-bc97-4082-8527-cc5473d03a02","Type":"ContainerStarted","Data":"343c3ae8e2083e57d5f79d2c54f626981b7519a4045d4b2d499fdb1feeee5a31"}
Jul 22 02:20:53.801266 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:53.801241    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4" event={"ID":"665c3881-a42c-4856-a81a-d85782af2571","Type":"ContainerStarted","Data":"86ca84e0bced468c19afa8a7e2df935e76695f5fe5eba41ce08a4cf48bae5885"}
Jul 22 02:20:53.811925 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:53.811900    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r7b6z" event={"ID":"9a07ecaa-d731-485d-a2d5-b28a59e7dee4","Type":"ContainerStarted","Data":"79c1c52a8dced3d9bac24e20b976cc4a0b25682a5ccb99d78a01c9a355f689fa"}
Jul 22 02:20:53.817684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:53.817603    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pnw2f" event={"ID":"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b","Type":"ContainerStarted","Data":"2f6ab52ee4867d2448ede1b6641e7c0e5bc1cdf76bb551b9719ed1f00ca39757"}
Jul 22 02:20:53.834107 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:53.834052    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal" event={"ID":"a0b30ae5bc27eeb039f1a3250327b5a6","Type":"ContainerStarted","Data":"aa08f2fd787a80a2a52330c2e1283de2ccb2d4bd4ccf21b825308f33a2444780"}
Jul 22 02:20:54.720785 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:54.720749    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:20:54.720973 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:54.720893    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4hdg" podUID="0036981e-4941-42f3-91d1-d1c98ffa1f5e"
Jul 22 02:20:54.721389 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:54.721320    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:20:54.721757 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:54.721426    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8jzv" podUID="ba53539c-579d-43a1-a6d0-d9020f0d18f3"
Jul 22 02:20:54.898545 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:54.898510    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal" event={"ID":"9e2f093a36821ff3dcaa0f49fa074849","Type":"ContainerStarted","Data":"90e636b37c1619935287a2df8ab2d583935d7d4f50b993838ef12a07c0b6a315"}
Jul 22 02:20:54.915297 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:54.915204    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-200.ec2.internal" podStartSLOduration=4.915184235 podStartE2EDuration="4.915184235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:20:53.851042081 +0000 UTC m=+4.699378657" watchObservedRunningTime="2026-07-22 02:20:54.915184235 +0000 UTC m=+5.763520811"
Jul 22 02:20:55.216002 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:55.215371    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret\") pod \"global-pull-secret-syncer-kzv8x\" (UID: \"faebb5f3-7db6-4e1c-88b0-a4530442e728\") " pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:20:55.216002 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:55.215529    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 02:20:55.216002 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:55.215594    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret podName:faebb5f3-7db6-4e1c-88b0-a4530442e728 nodeName:}" failed. No retries permitted until 2026-07-22 02:20:59.215575249 +0000 UTC m=+10.063911815 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret") pod "global-pull-secret-syncer-kzv8x" (UID: "faebb5f3-7db6-4e1c-88b0-a4530442e728") : object "kube-system"/"original-pull-secret" not registered
Jul 22 02:20:55.317157 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:55.316541    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs\") pod \"network-metrics-daemon-p8jzv\" (UID: \"ba53539c-579d-43a1-a6d0-d9020f0d18f3\") " pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:20:55.317157 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:55.316734    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:20:55.317157 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:55.316799    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs podName:ba53539c-579d-43a1-a6d0-d9020f0d18f3 nodeName:}" failed. No retries permitted until 2026-07-22 02:20:59.31677867 +0000 UTC m=+10.165115245 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs") pod "network-metrics-daemon-p8jzv" (UID: "ba53539c-579d-43a1-a6d0-d9020f0d18f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:20:55.417945 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:55.417908    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmgc\" (UniqueName: \"kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc\") pod \"network-check-target-r4hdg\" (UID: \"0036981e-4941-42f3-91d1-d1c98ffa1f5e\") " pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:20:55.418111 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:55.418093    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 02:20:55.418164 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:55.418115    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 02:20:55.418164 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:55.418128    2565 projected.go:196] Error preparing data for projected volume kube-api-access-sdmgc for pod openshift-network-diagnostics/network-check-target-r4hdg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:20:55.418289 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:55.418184    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc podName:0036981e-4941-42f3-91d1-d1c98ffa1f5e nodeName:}" failed. No retries permitted until 2026-07-22 02:20:59.418165888 +0000 UTC m=+10.266502463 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sdmgc" (UniqueName: "kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc") pod "network-check-target-r4hdg" (UID: "0036981e-4941-42f3-91d1-d1c98ffa1f5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:20:55.721383 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:55.721325    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:20:55.721558 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:55.721458    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzv8x" podUID="faebb5f3-7db6-4e1c-88b0-a4530442e728"
Jul 22 02:20:56.720253 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:56.720131    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:20:56.720253 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:56.720174    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:20:56.720500 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:56.720273    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8jzv" podUID="ba53539c-579d-43a1-a6d0-d9020f0d18f3"
Jul 22 02:20:56.720731 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:56.720686    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4hdg" podUID="0036981e-4941-42f3-91d1-d1c98ffa1f5e"
Jul 22 02:20:57.720706 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:57.720040    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:20:57.720706 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:57.720170    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzv8x" podUID="faebb5f3-7db6-4e1c-88b0-a4530442e728"
Jul 22 02:20:58.720563 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:58.720525    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:20:58.720734 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:58.720686    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8jzv" podUID="ba53539c-579d-43a1-a6d0-d9020f0d18f3"
Jul 22 02:20:58.721193 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:58.720526    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:20:58.721193 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:58.721014    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4hdg" podUID="0036981e-4941-42f3-91d1-d1c98ffa1f5e"
Jul 22 02:20:59.252137 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:59.252101    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret\") pod \"global-pull-secret-syncer-kzv8x\" (UID: \"faebb5f3-7db6-4e1c-88b0-a4530442e728\") " pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:20:59.252313 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:59.252242    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 02:20:59.252313 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:59.252306    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret podName:faebb5f3-7db6-4e1c-88b0-a4530442e728 nodeName:}" failed. No retries permitted until 2026-07-22 02:21:07.252287897 +0000 UTC m=+18.100624456 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret") pod "global-pull-secret-syncer-kzv8x" (UID: "faebb5f3-7db6-4e1c-88b0-a4530442e728") : object "kube-system"/"original-pull-secret" not registered
Jul 22 02:20:59.353107 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:59.353067    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs\") pod \"network-metrics-daemon-p8jzv\" (UID: \"ba53539c-579d-43a1-a6d0-d9020f0d18f3\") " pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:20:59.353275 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:59.353225    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:20:59.353345 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:59.353299    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs podName:ba53539c-579d-43a1-a6d0-d9020f0d18f3 nodeName:}" failed. No retries permitted until 2026-07-22 02:21:07.35327846 +0000 UTC m=+18.201615029 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs") pod "network-metrics-daemon-p8jzv" (UID: "ba53539c-579d-43a1-a6d0-d9020f0d18f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:20:59.453552 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:59.453512    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmgc\" (UniqueName: \"kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc\") pod \"network-check-target-r4hdg\" (UID: \"0036981e-4941-42f3-91d1-d1c98ffa1f5e\") " pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:20:59.453713 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:59.453687    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 02:20:59.453713 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:59.453706    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 02:20:59.453833 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:59.453718    2565 projected.go:196] Error preparing data for projected volume kube-api-access-sdmgc for pod openshift-network-diagnostics/network-check-target-r4hdg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:20:59.453833 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:59.453774    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc podName:0036981e-4941-42f3-91d1-d1c98ffa1f5e nodeName:}" failed. No retries permitted until 2026-07-22 02:21:07.45375628 +0000 UTC m=+18.302092847 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sdmgc" (UniqueName: "kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc") pod "network-check-target-r4hdg" (UID: "0036981e-4941-42f3-91d1-d1c98ffa1f5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:20:59.723752 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:20:59.723363    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:20:59.723752 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:20:59.723473    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzv8x" podUID="faebb5f3-7db6-4e1c-88b0-a4530442e728"
Jul 22 02:21:00.720906 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:00.720398    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:21:00.720906 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:00.720540    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8jzv" podUID="ba53539c-579d-43a1-a6d0-d9020f0d18f3"
Jul 22 02:21:00.720906 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:00.720758    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:21:00.721211 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:00.720928    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4hdg" podUID="0036981e-4941-42f3-91d1-d1c98ffa1f5e"
Jul 22 02:21:01.720495 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:01.720447    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:21:01.721032 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:01.720765    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzv8x" podUID="faebb5f3-7db6-4e1c-88b0-a4530442e728"
Jul 22 02:21:02.719859 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:02.719804    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:21:02.720045 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:02.719864    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:21:02.720045 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:02.719978    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8jzv" podUID="ba53539c-579d-43a1-a6d0-d9020f0d18f3"
Jul 22 02:21:02.720159 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:02.720118    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4hdg" podUID="0036981e-4941-42f3-91d1-d1c98ffa1f5e"
Jul 22 02:21:03.720743 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:03.720663    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:21:03.721194 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:03.720794    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzv8x" podUID="faebb5f3-7db6-4e1c-88b0-a4530442e728"
Jul 22 02:21:04.720391 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:04.720362    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:21:04.720584 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:04.720494    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4hdg" podUID="0036981e-4941-42f3-91d1-d1c98ffa1f5e"
Jul 22 02:21:04.720584 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:04.720545    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:21:04.720700 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:04.720680    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8jzv" podUID="ba53539c-579d-43a1-a6d0-d9020f0d18f3"
Jul 22 02:21:05.719974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:05.719943    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:21:05.720392 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:05.720072    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzv8x" podUID="faebb5f3-7db6-4e1c-88b0-a4530442e728"
Jul 22 02:21:06.719795 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:06.719767    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:21:06.720028 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:06.719765    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:21:06.720028 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:06.719913    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8jzv" podUID="ba53539c-579d-43a1-a6d0-d9020f0d18f3"
Jul 22 02:21:06.720028 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:06.719935    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4hdg" podUID="0036981e-4941-42f3-91d1-d1c98ffa1f5e"
Jul 22 02:21:07.316207 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:07.316177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret\") pod \"global-pull-secret-syncer-kzv8x\" (UID: \"faebb5f3-7db6-4e1c-88b0-a4530442e728\") " pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:21:07.316361 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:07.316287    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 02:21:07.316361 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:07.316333    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret podName:faebb5f3-7db6-4e1c-88b0-a4530442e728 nodeName:}" failed. No retries permitted until 2026-07-22 02:21:23.316320286 +0000 UTC m=+34.164656838 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret") pod "global-pull-secret-syncer-kzv8x" (UID: "faebb5f3-7db6-4e1c-88b0-a4530442e728") : object "kube-system"/"original-pull-secret" not registered
Jul 22 02:21:07.417447 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:07.417415    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs\") pod \"network-metrics-daemon-p8jzv\" (UID: \"ba53539c-579d-43a1-a6d0-d9020f0d18f3\") " pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:21:07.417630 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:07.417582    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:21:07.417683 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:07.417667    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs podName:ba53539c-579d-43a1-a6d0-d9020f0d18f3 nodeName:}" failed. No retries permitted until 2026-07-22 02:21:23.417646541 +0000 UTC m=+34.265983092 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs") pod "network-metrics-daemon-p8jzv" (UID: "ba53539c-579d-43a1-a6d0-d9020f0d18f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:21:07.518816 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:07.518780    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmgc\" (UniqueName: \"kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc\") pod \"network-check-target-r4hdg\" (UID: \"0036981e-4941-42f3-91d1-d1c98ffa1f5e\") " pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:21:07.519036 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:07.518954    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 02:21:07.519036 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:07.518970    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 02:21:07.519036 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:07.518980    2565 projected.go:196] Error preparing data for projected volume kube-api-access-sdmgc for pod openshift-network-diagnostics/network-check-target-r4hdg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:21:07.519169 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:07.519041    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc podName:0036981e-4941-42f3-91d1-d1c98ffa1f5e nodeName:}" failed. No retries permitted until 2026-07-22 02:21:23.519025837 +0000 UTC m=+34.367362390 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sdmgc" (UniqueName: "kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc") pod "network-check-target-r4hdg" (UID: "0036981e-4941-42f3-91d1-d1c98ffa1f5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:21:07.720409 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:07.720335    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:21:07.720876 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:07.720470    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzv8x" podUID="faebb5f3-7db6-4e1c-88b0-a4530442e728"
Jul 22 02:21:08.719986 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:08.719954    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:21:08.720169 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:08.719956    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:21:08.720169 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:08.720081    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4hdg" podUID="0036981e-4941-42f3-91d1-d1c98ffa1f5e"
Jul 22 02:21:08.720288 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:08.720176    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8jzv" podUID="ba53539c-579d-43a1-a6d0-d9020f0d18f3"
Jul 22 02:21:09.721536 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:09.721492    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:21:09.722013 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:09.721608    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzv8x" podUID="faebb5f3-7db6-4e1c-88b0-a4530442e728"
Jul 22 02:21:10.720506 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:10.720474    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:21:10.720666 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:10.720474    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:21:10.720666 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:10.720597    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4hdg" podUID="0036981e-4941-42f3-91d1-d1c98ffa1f5e"
Jul 22 02:21:10.720785 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:10.720691    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8jzv" podUID="ba53539c-579d-43a1-a6d0-d9020f0d18f3"
Jul 22 02:21:11.720471 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:11.720289    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:21:11.720974 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:11.720556    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzv8x" podUID="faebb5f3-7db6-4e1c-88b0-a4530442e728"
Jul 22 02:21:11.933172 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:11.933090    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zhtxk" event={"ID":"f5be685a-0b0f-4058-9272-2fb16817f84d","Type":"ContainerStarted","Data":"cbc7ac58e973b27616e70386f0ece624c13ade009f92aea549527a7bfe52d9ab"}
Jul 22 02:21:11.936115 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:11.936092    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jkrrj" event={"ID":"41818c5b-aa1c-42c2-ab5d-9d37d9a8e23b","Type":"ContainerStarted","Data":"b190985d9b54ce2ffbfeb317828132f5ff6ae94c4ef1d16b6cc68c4b840b4ec2"}
Jul 22 02:21:11.937442 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:11.937422    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7" event={"ID":"4f5729d9-bc97-4082-8527-cc5473d03a02","Type":"ContainerStarted","Data":"3cbaa4b4a0bc55889cd6e958f925d328ea647284e2e0f9e3069183046e27ae7c"}
Jul 22 02:21:11.939284 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:11.939263    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4" event={"ID":"665c3881-a42c-4856-a81a-d85782af2571","Type":"ContainerStarted","Data":"1f99f37615de39d8d7c8c54203eaed9c0f8f543209c8703fd23ee973dc7ea052"}
Jul 22 02:21:11.939367 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:11.939327    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4" event={"ID":"665c3881-a42c-4856-a81a-d85782af2571","Type":"ContainerStarted","Data":"4a188125d489d6af3a37fccddf71a063a85126ed900e1967d9c13a436b711bad"}
Jul 22 02:21:11.939367 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:11.939357    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4" event={"ID":"665c3881-a42c-4856-a81a-d85782af2571","Type":"ContainerStarted","Data":"5fb83278c5ea5658ec53b2a95e17eee28656e297ef73afb2080a794da0b49a9c"}
Jul 22 02:21:11.940492 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:11.940468    2565 generic.go:356] "Generic (PLEG): container finished" podID="32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b" containerID="807c17f2cb1cdc8a38edc7280da047a8357854ef9716d198d00368104528b5d7" exitCode=0
Jul 22 02:21:11.940564 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:11.940544    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pnw2f" event={"ID":"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b","Type":"ContainerDied","Data":"807c17f2cb1cdc8a38edc7280da047a8357854ef9716d198d00368104528b5d7"}
Jul 22 02:21:11.942201 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:11.942171    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kvk2x" event={"ID":"e736b988-93b9-40f9-9ab3-e3ee1a72f5a8","Type":"ContainerStarted","Data":"8617ec01e1cb454b1ba66cbeee525b65b52a04af93998d95403a00d79c42e592"}
Jul 22 02:21:11.943626 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:11.943596    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v6sck" event={"ID":"a1a137a5-3d05-45cf-8620-7537635e87d0","Type":"ContainerStarted","Data":"8cd3d6e878ceec8af016f9207500baf7c9c8abfb650a10db8f9fecf66f869d7e"}
Jul 22 02:21:11.945071 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:11.945050    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4ggf4" event={"ID":"d9824be5-cf4c-49b5-9ad4-db70c783d24c","Type":"ContainerStarted","Data":"a11abab2cc3aa7aae46b78c6697d11edff4836b4fd0417295e6c645780b0e8fe"}
Jul 22 02:21:11.947766 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:11.947730    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zhtxk" podStartSLOduration=4.444371056 podStartE2EDuration="22.947718701s" totalImagesPullingTime="18.503347645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:11.947321632 +0000 UTC m=+22.795658208" watchObservedRunningTime="2026-07-22 02:21:11.947718701 +0000 UTC m=+22.796055275"
Jul 22 02:21:11.947843 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:11.947823    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-200.ec2.internal" podStartSLOduration=21.947817278 podStartE2EDuration="21.947817278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:20:54.915947739 +0000 UTC m=+5.764284309" watchObservedRunningTime="2026-07-22 02:21:11.947817278 +0000 UTC m=+22.796153863"
Jul 22 02:21:11.961152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:11.961117    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v6sck" podStartSLOduration=4.455816514 podStartE2EDuration="22.961106691s" totalImagesPullingTime="18.505290177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:11.960710898 +0000 UTC m=+22.809047472" watchObservedRunningTime="2026-07-22 02:21:11.961106691 +0000 UTC m=+22.809443264"
Jul 22 02:21:11.976703 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:11.976673    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4ggf4" podStartSLOduration=4.470043887 podStartE2EDuration="22.97666102s" totalImagesPullingTime="18.506617133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:11.976523181 +0000 UTC m=+22.824859755" watchObservedRunningTime="2026-07-22 02:21:11.97666102 +0000 UTC m=+22.824997593"
Jul 22 02:21:11.993211 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:11.993181    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kvk2x" podStartSLOduration=4.44274659 podStartE2EDuration="22.993171103s" totalImagesPullingTime="18.550424513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:11.992920243 +0000 UTC m=+22.841256816" watchObservedRunningTime="2026-07-22 02:21:11.993171103 +0000 UTC m=+22.841507678"
Jul 22 02:21:12.720080 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:12.720017    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:21:12.720291 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:12.720133    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8jzv" podUID="ba53539c-579d-43a1-a6d0-d9020f0d18f3"
Jul 22 02:21:12.720291 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:12.720180    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:21:12.720291 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:12.720260    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4hdg" podUID="0036981e-4941-42f3-91d1-d1c98ffa1f5e"
Jul 22 02:21:12.950684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:12.950654    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4" event={"ID":"665c3881-a42c-4856-a81a-d85782af2571","Type":"ContainerStarted","Data":"a00c2033957e713fd5f5b3994e68ca4294fa0422bde3a1762f31d49db1bbffcb"}
Jul 22 02:21:12.951082 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:12.950743    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4" event={"ID":"665c3881-a42c-4856-a81a-d85782af2571","Type":"ContainerStarted","Data":"4fdd6d53b51b4e398fc082674db4a2b7365d794e6592be8899d99dc576e6b430"}
Jul 22 02:21:12.951082 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:12.950775    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4" event={"ID":"665c3881-a42c-4856-a81a-d85782af2571","Type":"ContainerStarted","Data":"6e033a13a698452990ea58f6a11a0180763ddd71bec9e0c9e42b3a10e9469e8f"}
Jul 22 02:21:12.952094 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:12.952067    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r7b6z" event={"ID":"9a07ecaa-d731-485d-a2d5-b28a59e7dee4","Type":"ContainerStarted","Data":"78d5ddef9402f890b148ec1252896c1b53ceef77a061f3dc138cd538ce7c6d14"}
Jul 22 02:21:12.956397 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:12.956376    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 22 02:21:12.970277 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:12.970208    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r7b6z" podStartSLOduration=5.461474813 podStartE2EDuration="23.970182635s" totalImagesPullingTime="18.508707822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:12.97017745 +0000 UTC m=+23.818514006" watchObservedRunningTime="2026-07-22 02:21:12.970182635 +0000 UTC m=+23.818519205"
Jul 22 02:21:12.970531 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:12.970501    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jkrrj" podStartSLOduration=5.468296977 podStartE2EDuration="23.970492649s" totalImagesPullingTime="18.502195672s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:12.029199818 +0000 UTC m=+22.877536394" watchObservedRunningTime="2026-07-22 02:21:12.970492649 +0000 UTC m=+23.818829224"
Jul 22 02:21:13.444271 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:13.444217    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4ggf4"
Jul 22 02:21:13.444863 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:13.444825    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4ggf4"
Jul 22 02:21:13.661931 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:13.661800    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T02:21:12.956391859Z","UUID":"030d6889-908d-4800-a0a3-f40a86dcc973","Handler":null,"Name":"","Endpoint":""}
Jul 22 02:21:13.663093 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:13.663071    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 22 02:21:13.663218 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:13.663114    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 22 02:21:13.720510 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:13.720310    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:21:13.720510 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:13.720447    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzv8x" podUID="faebb5f3-7db6-4e1c-88b0-a4530442e728"
Jul 22 02:21:13.957843 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:13.957810    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7" event={"ID":"4f5729d9-bc97-4082-8527-cc5473d03a02","Type":"ContainerStarted","Data":"865181f6510d7c1c11e22108a775d8be89be03b3e0046682161909c7bba5abac"}
Jul 22 02:21:13.958292 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:13.958074    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4ggf4"
Jul 22 02:21:13.958504 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:13.958487    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4ggf4"
Jul 22 02:21:14.719997 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:14.719803    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:21:14.720243 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:14.719827    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:21:14.720243 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:14.720064    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4hdg" podUID="0036981e-4941-42f3-91d1-d1c98ffa1f5e"
Jul 22 02:21:14.720243 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:14.720140    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8jzv" podUID="ba53539c-579d-43a1-a6d0-d9020f0d18f3"
Jul 22 02:21:14.963256 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:14.963218    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7" event={"ID":"4f5729d9-bc97-4082-8527-cc5473d03a02","Type":"ContainerStarted","Data":"c35da90aca7d3a513f05bb7c6539dc5255c071ef337ff1a2947a2ca8da4cc737"}
Jul 22 02:21:14.966820 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:14.966791    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4" event={"ID":"665c3881-a42c-4856-a81a-d85782af2571","Type":"ContainerStarted","Data":"cdcc9572a2122c4e59518f95fdbb187517a9a4de9f4bbde70434714e295c4c71"}
Jul 22 02:21:14.982713 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:14.982630    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lmph7" podStartSLOduration=5.186377236 podStartE2EDuration="25.982611763s" totalImagesPullingTime="20.796234527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:20:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:14.982254298 +0000 UTC m=+25.830590872" watchObservedRunningTime="2026-07-22 02:21:14.982611763 +0000 UTC m=+25.830948339"
Jul 22 02:21:15.720561 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:15.720533    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:21:15.720769 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:15.720662    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzv8x" podUID="faebb5f3-7db6-4e1c-88b0-a4530442e728"
Jul 22 02:21:16.720314 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:16.719766    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:21:16.720314 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:16.719774    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:21:16.720314 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:16.719878    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4hdg" podUID="0036981e-4941-42f3-91d1-d1c98ffa1f5e"
Jul 22 02:21:16.720314 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:16.719925    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8jzv" podUID="ba53539c-579d-43a1-a6d0-d9020f0d18f3"
Jul 22 02:21:16.974356 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:16.974275    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4" event={"ID":"665c3881-a42c-4856-a81a-d85782af2571","Type":"ContainerStarted","Data":"f4071d58601c36c8824747b703309a91b79d9cae57b17761d434557dc35257b4"}
Jul 22 02:21:16.974572 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:16.974553    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:21:16.976136 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:16.976110    2565 generic.go:356] "Generic (PLEG): container finished" podID="32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b" containerID="9df2c015da7e5f0161e6da5ef8e79571a0627bd2e4d712eafb8a0b94b2abb7cd" exitCode=0
Jul 22 02:21:16.976249 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:16.976148    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pnw2f" event={"ID":"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b","Type":"ContainerDied","Data":"9df2c015da7e5f0161e6da5ef8e79571a0627bd2e4d712eafb8a0b94b2abb7cd"}
Jul 22 02:21:16.989360 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:16.989339    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:21:17.005282 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:17.005247    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4" podStartSLOduration=9.382200428 podStartE2EDuration="28.005233333s" totalImagesPullingTime="18.623032905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:17.004802789 +0000 UTC m=+27.853139365" watchObservedRunningTime="2026-07-22 02:21:17.005233333 +0000 UTC m=+27.853569907"
Jul 22 02:21:17.720957 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:17.720546    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:21:17.720957 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:17.720644    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzv8x" podUID="faebb5f3-7db6-4e1c-88b0-a4530442e728"
Jul 22 02:21:17.867182 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:17.867151    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:17.965761 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:17.965702    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:17.978551 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:17.978499    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:21:17.978674 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:17.978578    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:21:17.993201 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:17.993181    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:21:18.688589 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:18.688558    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kzv8x"]
Jul 22 02:21:18.688807 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:18.688666    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:21:18.688807 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:18.688771    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzv8x" podUID="faebb5f3-7db6-4e1c-88b0-a4530442e728"
Jul 22 02:21:18.694814 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:18.694779    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r4hdg"]
Jul 22 02:21:18.694952 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:18.694905    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:21:18.695010 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:18.694988    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4hdg" podUID="0036981e-4941-42f3-91d1-d1c98ffa1f5e"
Jul 22 02:21:18.695684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:18.695658    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p8jzv"]
Jul 22 02:21:18.695786 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:18.695767    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:21:18.695876 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:18.695842    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8jzv" podUID="ba53539c-579d-43a1-a6d0-d9020f0d18f3"
Jul 22 02:21:18.981865 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:18.981816    2565 generic.go:356] "Generic (PLEG): container finished" podID="32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b" containerID="da25aa30e54d8aaaf752a68b3eff6c4faff306a7f09807407ab394887e2c9a0d" exitCode=0
Jul 22 02:21:18.982259 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:18.981904    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pnw2f" event={"ID":"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b","Type":"ContainerDied","Data":"da25aa30e54d8aaaf752a68b3eff6c4faff306a7f09807407ab394887e2c9a0d"}
Jul 22 02:21:19.051610 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:19.051589    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:19.194349 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:19.194321    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:20.720367 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:20.720281    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:21:20.720788 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:20.720382    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:21:20.720788 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:20.720389    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4hdg" podUID="0036981e-4941-42f3-91d1-d1c98ffa1f5e"
Jul 22 02:21:20.720788 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:20.720409    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:21:20.720788 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:20.720486    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8jzv" podUID="ba53539c-579d-43a1-a6d0-d9020f0d18f3"
Jul 22 02:21:20.720788 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:20.720539    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzv8x" podUID="faebb5f3-7db6-4e1c-88b0-a4530442e728"
Jul 22 02:21:20.988647 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:20.988579    2565 generic.go:356] "Generic (PLEG): container finished" podID="32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b" containerID="0647d2b5517625829fdd4b13e5d8f0449d0fbcd957115cc057807bafe7d0846d" exitCode=0
Jul 22 02:21:20.988647 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:20.988637    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pnw2f" event={"ID":"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b","Type":"ContainerDied","Data":"0647d2b5517625829fdd4b13e5d8f0449d0fbcd957115cc057807bafe7d0846d"}
Jul 22 02:21:22.084742 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:22.084709    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:22.160373 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:22.160340    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:22.261309 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:22.261277    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:22.327653 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:22.327625    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:22.391107 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:22.391079    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:22.457168 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:22.457134    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:22.720501 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:22.720428    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:21:22.720650 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:22.720539    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:21:22.720650 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:22.720544    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4hdg" podUID="0036981e-4941-42f3-91d1-d1c98ffa1f5e"
Jul 22 02:21:22.720759 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:22.720660    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8jzv" podUID="ba53539c-579d-43a1-a6d0-d9020f0d18f3"
Jul 22 02:21:22.720759 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:22.720547    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:21:22.720873 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:22.720804    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzv8x" podUID="faebb5f3-7db6-4e1c-88b0-a4530442e728"
Jul 22 02:21:23.349947 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:23.349863    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret\") pod \"global-pull-secret-syncer-kzv8x\" (UID: \"faebb5f3-7db6-4e1c-88b0-a4530442e728\") " pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:21:23.350459 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:23.350001    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 02:21:23.350459 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:23.350064    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret podName:faebb5f3-7db6-4e1c-88b0-a4530442e728 nodeName:}" failed. No retries permitted until 2026-07-22 02:21:55.350044927 +0000 UTC m=+66.198381482 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret") pod "global-pull-secret-syncer-kzv8x" (UID: "faebb5f3-7db6-4e1c-88b0-a4530442e728") : object "kube-system"/"original-pull-secret" not registered
Jul 22 02:21:23.450593 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:23.450554    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs\") pod \"network-metrics-daemon-p8jzv\" (UID: \"ba53539c-579d-43a1-a6d0-d9020f0d18f3\") " pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:21:23.450764 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:23.450715    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:21:23.450823 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:23.450784    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs podName:ba53539c-579d-43a1-a6d0-d9020f0d18f3 nodeName:}" failed. No retries permitted until 2026-07-22 02:21:55.450764548 +0000 UTC m=+66.299101105 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs") pod "network-metrics-daemon-p8jzv" (UID: "ba53539c-579d-43a1-a6d0-d9020f0d18f3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:21:23.551302 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:23.551269    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmgc\" (UniqueName: \"kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc\") pod \"network-check-target-r4hdg\" (UID: \"0036981e-4941-42f3-91d1-d1c98ffa1f5e\") " pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:21:23.551481 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:23.551396    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 02:21:23.551481 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:23.551415    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 02:21:23.551481 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:23.551425    2565 projected.go:196] Error preparing data for projected volume kube-api-access-sdmgc for pod openshift-network-diagnostics/network-check-target-r4hdg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:21:23.551481 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:23.551477    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc podName:0036981e-4941-42f3-91d1-d1c98ffa1f5e nodeName:}" failed. No retries permitted until 2026-07-22 02:21:55.551459584 +0000 UTC m=+66.399796137 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sdmgc" (UniqueName: "kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc") pod "network-check-target-r4hdg" (UID: "0036981e-4941-42f3-91d1-d1c98ffa1f5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:21:24.720283 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:24.720243    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:21:24.720283 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:24.720286    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:21:24.720972 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:24.720296    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:21:24.720972 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:24.720382    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kzv8x" podUID="faebb5f3-7db6-4e1c-88b0-a4530442e728"
Jul 22 02:21:24.720972 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:24.720479    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4hdg" podUID="0036981e-4941-42f3-91d1-d1c98ffa1f5e"
Jul 22 02:21:24.720972 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:24.720590    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8jzv" podUID="ba53539c-579d-43a1-a6d0-d9020f0d18f3"
Jul 22 02:21:25.460316 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.460293    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-200.ec2.internal" event="NodeReady"
Jul 22 02:21:25.460434 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.460425    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 22 02:21:25.493590 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.493565    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-566c7f66f7-9rsq9"]
Jul 22 02:21:25.495214 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.495199    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.497646 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.497627    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 22 02:21:25.497646 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.497637    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 22 02:21:25.497937 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.497923    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 22 02:21:25.498086 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.498074    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rvkph\""
Jul 22 02:21:25.503980 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.503937    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 22 02:21:25.505679 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.505658    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k2gpv"]
Jul 22 02:21:25.509641 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.509620    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-p6j8z"]
Jul 22 02:21:25.509751 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.509736    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:25.511330 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.511309    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p6j8z"
Jul 22 02:21:25.512116 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.512096    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jpzdk\""
Jul 22 02:21:25.512235 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.512114    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 22 02:21:25.512325 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.512100    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-566c7f66f7-9rsq9"]
Jul 22 02:21:25.512325 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.512285    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 22 02:21:25.513752 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.513734    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 22 02:21:25.513819 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.513735    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 22 02:21:25.513972 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.513957    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4hpjf\""
Jul 22 02:21:25.514024 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.513983    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 22 02:21:25.524193 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.524175    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k2gpv"]
Jul 22 02:21:25.524702 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.524684    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p6j8z"]
Jul 22 02:21:25.568918 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.568894    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-certificates\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.569038 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.568932    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-trusted-ca\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.569038 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.568950    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/417b06e9-0b5a-486a-92fa-aa9e118b7c36-config-volume\") pod \"dns-default-k2gpv\" (UID: \"417b06e9-0b5a-486a-92fa-aa9e118b7c36\") " pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:25.569038 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.568990    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-ca-trust-extracted\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.569038 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.569021    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-image-registry-private-configuration\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.569038 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.569039    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-installation-pull-secrets\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.569235 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.569067    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcjjp\" (UniqueName: \"kubernetes.io/projected/cfb4b1be-f309-43a2-8a40-b109ef938765-kube-api-access-jcjjp\") pod \"ingress-canary-p6j8z\" (UID: \"cfb4b1be-f309-43a2-8a40-b109ef938765\") " pod="openshift-ingress-canary/ingress-canary-p6j8z"
Jul 22 02:21:25.569235 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.569084    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfb4b1be-f309-43a2-8a40-b109ef938765-cert\") pod \"ingress-canary-p6j8z\" (UID: \"cfb4b1be-f309-43a2-8a40-b109ef938765\") " pod="openshift-ingress-canary/ingress-canary-p6j8z"
Jul 22 02:21:25.569235 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.569101    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-bound-sa-token\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.569235 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.569115    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l4x2\" (UniqueName: \"kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-kube-api-access-9l4x2\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.569235 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.569132    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.569235 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.569145    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/417b06e9-0b5a-486a-92fa-aa9e118b7c36-metrics-tls\") pod \"dns-default-k2gpv\" (UID: \"417b06e9-0b5a-486a-92fa-aa9e118b7c36\") " pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:25.569235 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.569160    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/417b06e9-0b5a-486a-92fa-aa9e118b7c36-tmp-dir\") pod \"dns-default-k2gpv\" (UID: \"417b06e9-0b5a-486a-92fa-aa9e118b7c36\") " pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:25.569235 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.569195    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz59p\" (UniqueName: \"kubernetes.io/projected/417b06e9-0b5a-486a-92fa-aa9e118b7c36-kube-api-access-bz59p\") pod \"dns-default-k2gpv\" (UID: \"417b06e9-0b5a-486a-92fa-aa9e118b7c36\") " pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:25.654788 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.654749    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:25.670000 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.669976    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-image-registry-private-configuration\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.670063 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.670005    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-installation-pull-secrets\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.670063 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.670035    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcjjp\" (UniqueName: \"kubernetes.io/projected/cfb4b1be-f309-43a2-8a40-b109ef938765-kube-api-access-jcjjp\") pod \"ingress-canary-p6j8z\" (UID: \"cfb4b1be-f309-43a2-8a40-b109ef938765\") " pod="openshift-ingress-canary/ingress-canary-p6j8z"
Jul 22 02:21:25.670063 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.670057    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfb4b1be-f309-43a2-8a40-b109ef938765-cert\") pod \"ingress-canary-p6j8z\" (UID: \"cfb4b1be-f309-43a2-8a40-b109ef938765\") " pod="openshift-ingress-canary/ingress-canary-p6j8z"
Jul 22 02:21:25.670202 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.670074    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-bound-sa-token\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.670202 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.670090    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l4x2\" (UniqueName: \"kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-kube-api-access-9l4x2\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.670202 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.670113    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.670202 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.670128    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/417b06e9-0b5a-486a-92fa-aa9e118b7c36-metrics-tls\") pod \"dns-default-k2gpv\" (UID: \"417b06e9-0b5a-486a-92fa-aa9e118b7c36\") " pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:25.670202 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.670148    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/417b06e9-0b5a-486a-92fa-aa9e118b7c36-tmp-dir\") pod \"dns-default-k2gpv\" (UID: \"417b06e9-0b5a-486a-92fa-aa9e118b7c36\") " pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:25.670432 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:25.670238    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 22 02:21:25.670432 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:25.670254    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 02:21:25.670432 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.670278    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz59p\" (UniqueName: \"kubernetes.io/projected/417b06e9-0b5a-486a-92fa-aa9e118b7c36-kube-api-access-bz59p\") pod \"dns-default-k2gpv\" (UID: \"417b06e9-0b5a-486a-92fa-aa9e118b7c36\") " pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:25.670432 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:25.670312    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/417b06e9-0b5a-486a-92fa-aa9e118b7c36-metrics-tls podName:417b06e9-0b5a-486a-92fa-aa9e118b7c36 nodeName:}" failed. No retries permitted until 2026-07-22 02:21:26.170293084 +0000 UTC m=+37.018629651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/417b06e9-0b5a-486a-92fa-aa9e118b7c36-metrics-tls") pod "dns-default-k2gpv" (UID: "417b06e9-0b5a-486a-92fa-aa9e118b7c36") : secret "dns-default-metrics-tls" not found
Jul 22 02:21:25.670432 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:25.670257    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-566c7f66f7-9rsq9: secret "image-registry-tls" not found
Jul 22 02:21:25.670432 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.670392    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-certificates\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.670432 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:25.670406    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls podName:9b9cc81f-d487-41fd-b3f4-8ae35600b8c5 nodeName:}" failed. No retries permitted until 2026-07-22 02:21:26.170387674 +0000 UTC m=+37.018724226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls") pod "image-registry-566c7f66f7-9rsq9" (UID: "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5") : secret "image-registry-tls" not found
Jul 22 02:21:25.670770 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.670458    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-trusted-ca\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.670770 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.670488    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/417b06e9-0b5a-486a-92fa-aa9e118b7c36-config-volume\") pod \"dns-default-k2gpv\" (UID: \"417b06e9-0b5a-486a-92fa-aa9e118b7c36\") " pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:25.670770 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.670517    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-ca-trust-extracted\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.670770 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.670461    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/417b06e9-0b5a-486a-92fa-aa9e118b7c36-tmp-dir\") pod \"dns-default-k2gpv\" (UID: \"417b06e9-0b5a-486a-92fa-aa9e118b7c36\") " pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:25.670982 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.670928    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-ca-trust-extracted\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.671074 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.671049    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-certificates\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.671074 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.671063    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/417b06e9-0b5a-486a-92fa-aa9e118b7c36-config-volume\") pod \"dns-default-k2gpv\" (UID: \"417b06e9-0b5a-486a-92fa-aa9e118b7c36\") " pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:25.671623 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.671603    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-trusted-ca\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.674254 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.674134    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-installation-pull-secrets\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.674316 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.674165    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfb4b1be-f309-43a2-8a40-b109ef938765-cert\") pod \"ingress-canary-p6j8z\" (UID: \"cfb4b1be-f309-43a2-8a40-b109ef938765\") " pod="openshift-ingress-canary/ingress-canary-p6j8z"
Jul 22 02:21:25.674316 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.674174    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-image-registry-private-configuration\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.680202 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.680164    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l4x2\" (UniqueName: \"kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-kube-api-access-9l4x2\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.680570 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.680549    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-bound-sa-token\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:25.680783 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.680765    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcjjp\" (UniqueName: \"kubernetes.io/projected/cfb4b1be-f309-43a2-8a40-b109ef938765-kube-api-access-jcjjp\") pod \"ingress-canary-p6j8z\" (UID: \"cfb4b1be-f309-43a2-8a40-b109ef938765\") " pod="openshift-ingress-canary/ingress-canary-p6j8z"
Jul 22 02:21:25.681993 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.681975    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz59p\" (UniqueName: \"kubernetes.io/projected/417b06e9-0b5a-486a-92fa-aa9e118b7c36-kube-api-access-bz59p\") pod \"dns-default-k2gpv\" (UID: \"417b06e9-0b5a-486a-92fa-aa9e118b7c36\") " pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:25.741383 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.741361    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:25.823147 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.823080    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:25.825044 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.825029    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p6j8z"
Jul 22 02:21:25.901025 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.901000    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:25.957887 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.957841    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p6j8z"]
Jul 22 02:21:25.961640 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:21:25.961613    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb4b1be_f309_43a2_8a40_b109ef938765.slice/crio-42ce7dc4c584312c1fa0be9bb3c44b61f5ee52c9219ad35ab4b508b9241bc048 WatchSource:0}: Error finding container 42ce7dc4c584312c1fa0be9bb3c44b61f5ee52c9219ad35ab4b508b9241bc048: Status 404 returned error can't find the container with id 42ce7dc4c584312c1fa0be9bb3c44b61f5ee52c9219ad35ab4b508b9241bc048
Jul 22 02:21:25.964564 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:25.964531    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:26.002913 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:26.002891    2565 generic.go:356] "Generic (PLEG): container finished" podID="32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b" containerID="c5c90751d098c77919975524cdb07ffeebd9cd78b4d98641ca34778c59a112ee" exitCode=0
Jul 22 02:21:26.003008 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:26.002957    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pnw2f" event={"ID":"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b","Type":"ContainerDied","Data":"c5c90751d098c77919975524cdb07ffeebd9cd78b4d98641ca34778c59a112ee"}
Jul 22 02:21:26.003991 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:26.003972    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p6j8z" event={"ID":"cfb4b1be-f309-43a2-8a40-b109ef938765","Type":"ContainerStarted","Data":"42ce7dc4c584312c1fa0be9bb3c44b61f5ee52c9219ad35ab4b508b9241bc048"}
Jul 22 02:21:26.174531 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:26.174504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:26.174627 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:26.174537    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/417b06e9-0b5a-486a-92fa-aa9e118b7c36-metrics-tls\") pod \"dns-default-k2gpv\" (UID: \"417b06e9-0b5a-486a-92fa-aa9e118b7c36\") " pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:26.174668 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:26.174638    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 02:21:26.174708 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:26.174686    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/417b06e9-0b5a-486a-92fa-aa9e118b7c36-metrics-tls podName:417b06e9-0b5a-486a-92fa-aa9e118b7c36 nodeName:}" failed. No retries permitted until 2026-07-22 02:21:27.17467293 +0000 UTC m=+38.023009485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/417b06e9-0b5a-486a-92fa-aa9e118b7c36-metrics-tls") pod "dns-default-k2gpv" (UID: "417b06e9-0b5a-486a-92fa-aa9e118b7c36") : secret "dns-default-metrics-tls" not found
Jul 22 02:21:26.174748 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:26.174642    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 22 02:21:26.174748 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:26.174730    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-566c7f66f7-9rsq9: secret "image-registry-tls" not found
Jul 22 02:21:26.174804 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:26.174774    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls podName:9b9cc81f-d487-41fd-b3f4-8ae35600b8c5 nodeName:}" failed. No retries permitted until 2026-07-22 02:21:27.174761022 +0000 UTC m=+38.023097582 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls") pod "image-registry-566c7f66f7-9rsq9" (UID: "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5") : secret "image-registry-tls" not found
Jul 22 02:21:26.719931 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:26.719888    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:21:26.719931 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:26.719929    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:21:26.720173 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:26.719937    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:21:26.722293 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:26.722273    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 22 02:21:26.722462 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:26.722442    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 22 02:21:26.722561 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:26.722541    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 22 02:21:26.722735 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:26.722710    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 22 02:21:26.722837 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:26.722739    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cw2sm\""
Jul 22 02:21:26.722837 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:26.722748    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8mll8\""
Jul 22 02:21:26.850884 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:26.850836    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:26.911510 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:26.911478    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:27.009622 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:27.009547    2565 generic.go:356] "Generic (PLEG): container finished" podID="32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b" containerID="b0e09b6528468dfa3891ab398f7097737943fece2420a2920647a088c3b774f6" exitCode=0
Jul 22 02:21:27.009622 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:27.009598    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pnw2f" event={"ID":"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b","Type":"ContainerDied","Data":"b0e09b6528468dfa3891ab398f7097737943fece2420a2920647a088c3b774f6"}
Jul 22 02:21:27.181203 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:27.181004    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:27.181345 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:27.181224    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/417b06e9-0b5a-486a-92fa-aa9e118b7c36-metrics-tls\") pod \"dns-default-k2gpv\" (UID: \"417b06e9-0b5a-486a-92fa-aa9e118b7c36\") " pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:27.181345 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:27.181183    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 22 02:21:27.181345 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:27.181285    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-566c7f66f7-9rsq9: secret "image-registry-tls" not found
Jul 22 02:21:27.181492 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:27.181342    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls podName:9b9cc81f-d487-41fd-b3f4-8ae35600b8c5 nodeName:}" failed. No retries permitted until 2026-07-22 02:21:29.181323092 +0000 UTC m=+40.029659650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls") pod "image-registry-566c7f66f7-9rsq9" (UID: "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5") : secret "image-registry-tls" not found
Jul 22 02:21:27.181492 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:27.181380    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 02:21:27.181492 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:27.181423    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/417b06e9-0b5a-486a-92fa-aa9e118b7c36-metrics-tls podName:417b06e9-0b5a-486a-92fa-aa9e118b7c36 nodeName:}" failed. No retries permitted until 2026-07-22 02:21:29.181408863 +0000 UTC m=+40.029745440 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/417b06e9-0b5a-486a-92fa-aa9e118b7c36-metrics-tls") pod "dns-default-k2gpv" (UID: "417b06e9-0b5a-486a-92fa-aa9e118b7c36") : secret "dns-default-metrics-tls" not found
Jul 22 02:21:28.016197 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:28.016151    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pnw2f" event={"ID":"32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b","Type":"ContainerStarted","Data":"94cdb32524563e46425feef11c139969e14b85b2ce8814c3d7c84f2a37283c1e"}
Jul 22 02:21:28.038919 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:28.038840    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pnw2f" podStartSLOduration=9.521701156 podStartE2EDuration="39.038821072s" totalImagesPullingTime="29.517119916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:20:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:28.03757242 +0000 UTC m=+38.885908996" watchObservedRunningTime="2026-07-22 02:21:28.038821072 +0000 UTC m=+38.887157649"
Jul 22 02:21:28.161103 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:28.161064    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:28.231780 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:28.231741    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:28.821190 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:28.821103    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:28.956441 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:28.956414    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:29.019865 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.019822    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p6j8z" event={"ID":"cfb4b1be-f309-43a2-8a40-b109ef938765","Type":"ContainerStarted","Data":"eb2bc38facf2ef64279af3c3d93716def61b4429ae9bee5b83ce7d815dfe747e"}
Jul 22 02:21:29.035232 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.035177    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-p6j8z" podStartSLOduration=1.5332215919999999 podStartE2EDuration="4.035158934s" totalImagesPullingTime="2.501937342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:29.034739848 +0000 UTC m=+39.883076421" watchObservedRunningTime="2026-07-22 02:21:29.035158934 +0000 UTC m=+39.883495512"
Jul 22 02:21:29.115110 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.115085    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:29.133647 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.133623    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xrb8s"]
Jul 22 02:21:29.154557 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.154529    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xrb8s"]
Jul 22 02:21:29.154654 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.154624    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xrb8s"
Jul 22 02:21:29.157164 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.157144    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 22 02:21:29.157274 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.157176    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 22 02:21:29.157368 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.157352    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-gp4tt\""
Jul 22 02:21:29.197037 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.197018    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:29.197152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.197045    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/417b06e9-0b5a-486a-92fa-aa9e118b7c36-metrics-tls\") pod \"dns-default-k2gpv\" (UID: \"417b06e9-0b5a-486a-92fa-aa9e118b7c36\") " pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:29.197152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.197106    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqmn7\" (UniqueName: \"kubernetes.io/projected/8e539c5a-a01e-4319-9135-774e86dc565f-kube-api-access-pqmn7\") pod \"migrator-7cc9c7ffd7-xrb8s\" (UID: \"8e539c5a-a01e-4319-9135-774e86dc565f\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xrb8s"
Jul 22 02:21:29.197258 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:29.197170    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 22 02:21:29.197258 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:29.197188    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-566c7f66f7-9rsq9: secret "image-registry-tls" not found
Jul 22 02:21:29.197258 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:29.197211    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 02:21:29.197258 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:29.197249    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls podName:9b9cc81f-d487-41fd-b3f4-8ae35600b8c5 nodeName:}" failed. No retries permitted until 2026-07-22 02:21:33.197230219 +0000 UTC m=+44.045566783 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls") pod "image-registry-566c7f66f7-9rsq9" (UID: "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5") : secret "image-registry-tls" not found
Jul 22 02:21:29.197452 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:29.197267    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/417b06e9-0b5a-486a-92fa-aa9e118b7c36-metrics-tls podName:417b06e9-0b5a-486a-92fa-aa9e118b7c36 nodeName:}" failed. No retries permitted until 2026-07-22 02:21:33.197257669 +0000 UTC m=+44.045594221 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/417b06e9-0b5a-486a-92fa-aa9e118b7c36-metrics-tls") pod "dns-default-k2gpv" (UID: "417b06e9-0b5a-486a-92fa-aa9e118b7c36") : secret "dns-default-metrics-tls" not found
Jul 22 02:21:29.208508 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.208482    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:29.297865 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.297824    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqmn7\" (UniqueName: \"kubernetes.io/projected/8e539c5a-a01e-4319-9135-774e86dc565f-kube-api-access-pqmn7\") pod \"migrator-7cc9c7ffd7-xrb8s\" (UID: \"8e539c5a-a01e-4319-9135-774e86dc565f\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xrb8s"
Jul 22 02:21:29.308216 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.308195    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqmn7\" (UniqueName: \"kubernetes.io/projected/8e539c5a-a01e-4319-9135-774e86dc565f-kube-api-access-pqmn7\") pod \"migrator-7cc9c7ffd7-xrb8s\" (UID: \"8e539c5a-a01e-4319-9135-774e86dc565f\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xrb8s"
Jul 22 02:21:29.308785 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.308749    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:29.408881 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.408791    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:29.463258 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.463227    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xrb8s"
Jul 22 02:21:29.608694 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.608658    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xrb8s"]
Jul 22 02:21:29.612447 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:21:29.612421    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e539c5a_a01e_4319_9135_774e86dc565f.slice/crio-6bd380a56e415e60f4777268dfd8d07c1f2597afb79a62c29ebc178c7b10fbb1 WatchSource:0}: Error finding container 6bd380a56e415e60f4777268dfd8d07c1f2597afb79a62c29ebc178c7b10fbb1: Status 404 returned error can't find the container with id 6bd380a56e415e60f4777268dfd8d07c1f2597afb79a62c29ebc178c7b10fbb1
Jul 22 02:21:29.613096 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.613079    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7984v"]
Jul 22 02:21:29.636099 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.636071    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:29.650243 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.650223    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7984v"]
Jul 22 02:21:29.650317 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.650307    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:29.652817 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.652798    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 22 02:21:29.652924 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.652817    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mzpsr\""
Jul 22 02:21:29.653037 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.653019    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 22 02:21:29.653099 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.653076    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 22 02:21:29.653656 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.653640    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 22 02:21:29.697703 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.697654    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:29.700676 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.700658    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/50921dd8-5d4d-4801-a4c5-339f81901b9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:29.700749 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.700689    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/50921dd8-5d4d-4801-a4c5-339f81901b9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:29.700749 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.700707    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhrcj\" (UniqueName: \"kubernetes.io/projected/50921dd8-5d4d-4801-a4c5-339f81901b9f-kube-api-access-mhrcj\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:29.700831 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.700771    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/50921dd8-5d4d-4801-a4c5-339f81901b9f-data-volume\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:29.700831 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.700803    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/50921dd8-5d4d-4801-a4c5-339f81901b9f-crio-socket\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:29.780545 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.780524    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:29.801498 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.801479    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/50921dd8-5d4d-4801-a4c5-339f81901b9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:29.801550 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.801504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrcj\" (UniqueName: \"kubernetes.io/projected/50921dd8-5d4d-4801-a4c5-339f81901b9f-kube-api-access-mhrcj\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:29.801550 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.801542    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/50921dd8-5d4d-4801-a4c5-339f81901b9f-data-volume\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:29.801624 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.801575    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/50921dd8-5d4d-4801-a4c5-339f81901b9f-crio-socket\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:29.801624 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:29.801610    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 02:21:29.801689 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.801626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/50921dd8-5d4d-4801-a4c5-339f81901b9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:29.801689 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:29.801665    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50921dd8-5d4d-4801-a4c5-339f81901b9f-insights-runtime-extractor-tls podName:50921dd8-5d4d-4801-a4c5-339f81901b9f nodeName:}" failed. No retries permitted until 2026-07-22 02:21:30.301646556 +0000 UTC m=+41.149983109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/50921dd8-5d4d-4801-a4c5-339f81901b9f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7984v" (UID: "50921dd8-5d4d-4801-a4c5-339f81901b9f") : secret "insights-runtime-extractor-tls" not found
Jul 22 02:21:29.801783 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:29.801702    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 22 02:21:29.801783 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:29.801756    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/50921dd8-5d4d-4801-a4c5-339f81901b9f-kube-rbac-proxy-cm podName:50921dd8-5d4d-4801-a4c5-339f81901b9f nodeName:}" failed. No retries permitted until 2026-07-22 02:21:30.301741812 +0000 UTC m=+41.150078365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/50921dd8-5d4d-4801-a4c5-339f81901b9f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7984v" (UID: "50921dd8-5d4d-4801-a4c5-339f81901b9f") : configmap "kube-rbac-proxy" not found
Jul 22 02:21:29.801876 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.801793    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/50921dd8-5d4d-4801-a4c5-339f81901b9f-crio-socket\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:29.801917 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.801902    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/50921dd8-5d4d-4801-a4c5-339f81901b9f-data-volume\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:29.810071 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.810054    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrcj\" (UniqueName: \"kubernetes.io/projected/50921dd8-5d4d-4801-a4c5-339f81901b9f-kube-api-access-mhrcj\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:29.884029 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:29.884005    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:30.023556 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:30.023481    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xrb8s" event={"ID":"8e539c5a-a01e-4319-9135-774e86dc565f","Type":"ContainerStarted","Data":"6bd380a56e415e60f4777268dfd8d07c1f2597afb79a62c29ebc178c7b10fbb1"}
Jul 22 02:21:30.044142 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:30.044115    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:30.156770 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:30.156746    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:30.304356 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:30.304278    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/50921dd8-5d4d-4801-a4c5-339f81901b9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:30.304356 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:30.304312    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/50921dd8-5d4d-4801-a4c5-339f81901b9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:30.304567 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:30.304406    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 22 02:21:30.304567 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:30.304483    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/50921dd8-5d4d-4801-a4c5-339f81901b9f-kube-rbac-proxy-cm podName:50921dd8-5d4d-4801-a4c5-339f81901b9f nodeName:}" failed. No retries permitted until 2026-07-22 02:21:31.304462651 +0000 UTC m=+42.152799218 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/50921dd8-5d4d-4801-a4c5-339f81901b9f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7984v" (UID: "50921dd8-5d4d-4801-a4c5-339f81901b9f") : configmap "kube-rbac-proxy" not found
Jul 22 02:21:30.304567 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:30.304415    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 02:21:30.304567 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:30.304534    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50921dd8-5d4d-4801-a4c5-339f81901b9f-insights-runtime-extractor-tls podName:50921dd8-5d4d-4801-a4c5-339f81901b9f nodeName:}" failed. No retries permitted until 2026-07-22 02:21:31.304523556 +0000 UTC m=+42.152860107 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/50921dd8-5d4d-4801-a4c5-339f81901b9f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7984v" (UID: "50921dd8-5d4d-4801-a4c5-339f81901b9f") : secret "insights-runtime-extractor-tls" not found
Jul 22 02:21:31.310980 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.310940    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/50921dd8-5d4d-4801-a4c5-339f81901b9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:31.310980 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.310985    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/50921dd8-5d4d-4801-a4c5-339f81901b9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:31.311408 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:31.311067    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 22 02:21:31.311408 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:31.311112    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 02:21:31.311408 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:31.311134    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/50921dd8-5d4d-4801-a4c5-339f81901b9f-kube-rbac-proxy-cm podName:50921dd8-5d4d-4801-a4c5-339f81901b9f nodeName:}" failed. No retries permitted until 2026-07-22 02:21:33.311116061 +0000 UTC m=+44.159452626 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/50921dd8-5d4d-4801-a4c5-339f81901b9f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7984v" (UID: "50921dd8-5d4d-4801-a4c5-339f81901b9f") : configmap "kube-rbac-proxy" not found
Jul 22 02:21:31.311408 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:31.311162    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50921dd8-5d4d-4801-a4c5-339f81901b9f-insights-runtime-extractor-tls podName:50921dd8-5d4d-4801-a4c5-339f81901b9f nodeName:}" failed. No retries permitted until 2026-07-22 02:21:33.311152604 +0000 UTC m=+44.159489156 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/50921dd8-5d4d-4801-a4c5-339f81901b9f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7984v" (UID: "50921dd8-5d4d-4801-a4c5-339f81901b9f") : secret "insights-runtime-extractor-tls" not found
Jul 22 02:21:31.445547 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.445518    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:31.545862 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.545816    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:31.638982 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.638958    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-qlppn"]
Jul 22 02:21:31.662217 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.662193    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-qlppn"]
Jul 22 02:21:31.662312 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.662298    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-qlppn"
Jul 22 02:21:31.664779 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.664758    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mzn5h\""
Jul 22 02:21:31.664902 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.664781    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 22 02:21:31.664902 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.664782    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 22 02:21:31.665669 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.665655    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 22 02:21:31.665932 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.665913    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 22 02:21:31.670568 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.670547    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:31.713878 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.713835    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv5zv\" (UniqueName: \"kubernetes.io/projected/0cc03024-70b3-42d1-9b78-335ab4c4bf0b-kube-api-access-rv5zv\") pod \"service-ca-58598695f7-qlppn\" (UID: \"0cc03024-70b3-42d1-9b78-335ab4c4bf0b\") " pod="openshift-service-ca/service-ca-58598695f7-qlppn"
Jul 22 02:21:31.713974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.713905    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0cc03024-70b3-42d1-9b78-335ab4c4bf0b-signing-cabundle\") pod \"service-ca-58598695f7-qlppn\" (UID: \"0cc03024-70b3-42d1-9b78-335ab4c4bf0b\") " pod="openshift-service-ca/service-ca-58598695f7-qlppn"
Jul 22 02:21:31.714026 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.713999    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0cc03024-70b3-42d1-9b78-335ab4c4bf0b-signing-key\") pod \"service-ca-58598695f7-qlppn\" (UID: \"0cc03024-70b3-42d1-9b78-335ab4c4bf0b\") " pod="openshift-service-ca/service-ca-58598695f7-qlppn"
Jul 22 02:21:31.769864 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.769824    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:31.814861 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.814827    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv5zv\" (UniqueName: \"kubernetes.io/projected/0cc03024-70b3-42d1-9b78-335ab4c4bf0b-kube-api-access-rv5zv\") pod \"service-ca-58598695f7-qlppn\" (UID: \"0cc03024-70b3-42d1-9b78-335ab4c4bf0b\") " pod="openshift-service-ca/service-ca-58598695f7-qlppn"
Jul 22 02:21:31.814967 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.814900    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0cc03024-70b3-42d1-9b78-335ab4c4bf0b-signing-cabundle\") pod \"service-ca-58598695f7-qlppn\" (UID: \"0cc03024-70b3-42d1-9b78-335ab4c4bf0b\") " pod="openshift-service-ca/service-ca-58598695f7-qlppn"
Jul 22 02:21:31.814967 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.814952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0cc03024-70b3-42d1-9b78-335ab4c4bf0b-signing-key\") pod \"service-ca-58598695f7-qlppn\" (UID: \"0cc03024-70b3-42d1-9b78-335ab4c4bf0b\") " pod="openshift-service-ca/service-ca-58598695f7-qlppn"
Jul 22 02:21:31.815519 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.815499    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0cc03024-70b3-42d1-9b78-335ab4c4bf0b-signing-cabundle\") pod \"service-ca-58598695f7-qlppn\" (UID: \"0cc03024-70b3-42d1-9b78-335ab4c4bf0b\") " pod="openshift-service-ca/service-ca-58598695f7-qlppn"
Jul 22 02:21:31.817305 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.817286    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0cc03024-70b3-42d1-9b78-335ab4c4bf0b-signing-key\") pod \"service-ca-58598695f7-qlppn\" (UID: \"0cc03024-70b3-42d1-9b78-335ab4c4bf0b\") " pod="openshift-service-ca/service-ca-58598695f7-qlppn"
Jul 22 02:21:31.824285 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.824268    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv5zv\" (UniqueName: \"kubernetes.io/projected/0cc03024-70b3-42d1-9b78-335ab4c4bf0b-kube-api-access-rv5zv\") pod \"service-ca-58598695f7-qlppn\" (UID: \"0cc03024-70b3-42d1-9b78-335ab4c4bf0b\") " pod="openshift-service-ca/service-ca-58598695f7-qlppn"
Jul 22 02:21:31.970826 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:31.970799    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-qlppn"
Jul 22 02:21:32.030620 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:32.030586    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xrb8s" event={"ID":"8e539c5a-a01e-4319-9135-774e86dc565f","Type":"ContainerStarted","Data":"07a13aa8500980cb74ce05bea86fec0aa7845687c359ba4fe595c671d045d9a3"}
Jul 22 02:21:32.030764 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:32.030642    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xrb8s" event={"ID":"8e539c5a-a01e-4319-9135-774e86dc565f","Type":"ContainerStarted","Data":"ef3b65fc8eb8e9572e80f3bef709f5974cba60b3f7632c6477d4dc188f021b31"}
Jul 22 02:21:32.051253 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:32.051202    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-xrb8s" podStartSLOduration=1.252939151 podStartE2EDuration="3.051186304s" totalImagesPullingTime="1.798247153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:32.049685593 +0000 UTC m=+42.898022169" watchObservedRunningTime="2026-07-22 02:21:32.051186304 +0000 UTC m=+42.899522878"
Jul 22 02:21:32.107644 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:32.107618    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-qlppn"]
Jul 22 02:21:32.110805 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:21:32.110780    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc03024_70b3_42d1_9b78_335ab4c4bf0b.slice/crio-517c4991cc2f211d905ad067ef8ace421073ea972c8e3d281972b41646c79f9a WatchSource:0}: Error finding container 517c4991cc2f211d905ad067ef8ace421073ea972c8e3d281972b41646c79f9a: Status 404 returned error can't find the container with id 517c4991cc2f211d905ad067ef8ace421073ea972c8e3d281972b41646c79f9a
Jul 22 02:21:32.140377 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:32.140351    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:32.180826 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:32.180808    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zhtxk_f5be685a-0b0f-4058-9272-2fb16817f84d/dns-node-resolver/0.log"
Jul 22 02:21:32.241746 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:32.241663    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:32.602679 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:32.602649    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:32.979274 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:32.979191    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v6sck_a1a137a5-3d05-45cf-8620-7537635e87d0/node-ca/0.log"
Jul 22 02:21:33.035066 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:33.035025    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-qlppn" event={"ID":"0cc03024-70b3-42d1-9b78-335ab4c4bf0b","Type":"ContainerStarted","Data":"517c4991cc2f211d905ad067ef8ace421073ea972c8e3d281972b41646c79f9a"}
Jul 22 02:21:33.225724 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:33.225686    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:33.225724 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:33.225722    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/417b06e9-0b5a-486a-92fa-aa9e118b7c36-metrics-tls\") pod \"dns-default-k2gpv\" (UID: \"417b06e9-0b5a-486a-92fa-aa9e118b7c36\") " pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:33.225915 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:33.225828    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 22 02:21:33.225915 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:33.225860    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-566c7f66f7-9rsq9: secret "image-registry-tls" not found
Jul 22 02:21:33.225915 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:33.225881    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 02:21:33.225915 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:33.225914    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls podName:9b9cc81f-d487-41fd-b3f4-8ae35600b8c5 nodeName:}" failed. No retries permitted until 2026-07-22 02:21:41.225899816 +0000 UTC m=+52.074236369 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls") pod "image-registry-566c7f66f7-9rsq9" (UID: "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5") : secret "image-registry-tls" not found
Jul 22 02:21:33.226051 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:33.225928    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/417b06e9-0b5a-486a-92fa-aa9e118b7c36-metrics-tls podName:417b06e9-0b5a-486a-92fa-aa9e118b7c36 nodeName:}" failed. No retries permitted until 2026-07-22 02:21:41.225921707 +0000 UTC m=+52.074258259 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/417b06e9-0b5a-486a-92fa-aa9e118b7c36-metrics-tls") pod "dns-default-k2gpv" (UID: "417b06e9-0b5a-486a-92fa-aa9e118b7c36") : secret "dns-default-metrics-tls" not found
Jul 22 02:21:33.326515 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:33.326433    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/50921dd8-5d4d-4801-a4c5-339f81901b9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:33.326515 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:33.326480    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/50921dd8-5d4d-4801-a4c5-339f81901b9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:33.326707 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:33.326580    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 22 02:21:33.326707 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:33.326662    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/50921dd8-5d4d-4801-a4c5-339f81901b9f-kube-rbac-proxy-cm podName:50921dd8-5d4d-4801-a4c5-339f81901b9f nodeName:}" failed. No retries permitted until 2026-07-22 02:21:37.326639374 +0000 UTC m=+48.174975930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/50921dd8-5d4d-4801-a4c5-339f81901b9f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7984v" (UID: "50921dd8-5d4d-4801-a4c5-339f81901b9f") : configmap "kube-rbac-proxy" not found
Jul 22 02:21:33.326707 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:33.326687    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 02:21:33.326863 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:33.326751    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50921dd8-5d4d-4801-a4c5-339f81901b9f-insights-runtime-extractor-tls podName:50921dd8-5d4d-4801-a4c5-339f81901b9f nodeName:}" failed. No retries permitted until 2026-07-22 02:21:37.326734445 +0000 UTC m=+48.175071001 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/50921dd8-5d4d-4801-a4c5-339f81901b9f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7984v" (UID: "50921dd8-5d4d-4801-a4c5-339f81901b9f") : secret "insights-runtime-extractor-tls" not found
Jul 22 02:21:33.725755 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:33.725723    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:33.777709 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:33.777677    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p6j8z_cfb4b1be-f309-43a2-8a40-b109ef938765/serve-healthcheck-canary/0.log"
Jul 22 02:21:33.978196 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:33.978110    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-xrb8s_8e539c5a-a01e-4319-9135-774e86dc565f/migrator/0.log"
Jul 22 02:21:33.981354 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:33.981328    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:34.081985 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:34.081952    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:34.176944 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:34.176918    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-xrb8s_8e539c5a-a01e-4319-9135-774e86dc565f/graceful-termination/0.log"
Jul 22 02:21:35.043085 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:35.043053    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-qlppn" event={"ID":"0cc03024-70b3-42d1-9b78-335ab4c4bf0b","Type":"ContainerStarted","Data":"ff6f59bc2d1efa1a09372f88b3f7b366405d2fa5f4b220530f037de21dc6660c"}
Jul 22 02:21:35.060214 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:35.060164    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-qlppn" podStartSLOduration=1.83538158 podStartE2EDuration="4.060146501s" totalImagesPullingTime="2.224764921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:35.05953662 +0000 UTC m=+45.907873197" watchObservedRunningTime="2026-07-22 02:21:35.060146501 +0000 UTC m=+45.908483075"
Jul 22 02:21:35.590007 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:35.589972    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:35.689903 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:35.689869    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:35.790367 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:35.790325    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:35.890416 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:35.890371    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:35.956752 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:35.956713    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:36.044013 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:36.043981    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:36.683572 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:36.683540    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:37.273464 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:37.273431    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:37.360160 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:37.360129    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/50921dd8-5d4d-4801-a4c5-339f81901b9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:37.360295 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:37.360279    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 02:21:37.360344 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:37.360316    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/50921dd8-5d4d-4801-a4c5-339f81901b9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:37.360391 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:21:37.360344    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50921dd8-5d4d-4801-a4c5-339f81901b9f-insights-runtime-extractor-tls podName:50921dd8-5d4d-4801-a4c5-339f81901b9f nodeName:}" failed. No retries permitted until 2026-07-22 02:21:45.360328108 +0000 UTC m=+56.208664660 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/50921dd8-5d4d-4801-a4c5-339f81901b9f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7984v" (UID: "50921dd8-5d4d-4801-a4c5-339f81901b9f") : secret "insights-runtime-extractor-tls" not found
Jul 22 02:21:37.360766 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:37.360743    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/50921dd8-5d4d-4801-a4c5-339f81901b9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:37.478527 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:37.478487    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:37.676140 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:37.676099    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:38.073819 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:38.073749    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:38.673521 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:38.673491    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:39.071420 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:39.071327    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:39.274686 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:39.274647    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:39.480496 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:39.480464    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:39.538836 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:39.538787    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:39.879906 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:39.879864    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:40.277171 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:40.277089    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:40.340738 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:40.340708    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:40.478697 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:40.478666    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:41.076087 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:41.076044    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:41.289039 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:41.289002    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:41.289039 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:41.289034    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/417b06e9-0b5a-486a-92fa-aa9e118b7c36-metrics-tls\") pod \"dns-default-k2gpv\" (UID: \"417b06e9-0b5a-486a-92fa-aa9e118b7c36\") " pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:41.291270 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:41.291247    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/417b06e9-0b5a-486a-92fa-aa9e118b7c36-metrics-tls\") pod \"dns-default-k2gpv\" (UID: \"417b06e9-0b5a-486a-92fa-aa9e118b7c36\") " pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:41.291418 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:41.291402    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls\") pod \"image-registry-566c7f66f7-9rsq9\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") " pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:41.404246 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:41.404219    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:41.419345 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:41.419316    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:41.474072 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:41.474008    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:41.565292 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:41.565258    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-566c7f66f7-9rsq9"]
Jul 22 02:21:41.570802 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:21:41.570773    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9cc81f_d487_41fd_b3f4_8ae35600b8c5.slice/crio-e1e26c1a7f1a7bea36621667a376501a049995ab2cd8c7f8bc2253ff5bdec908 WatchSource:0}: Error finding container e1e26c1a7f1a7bea36621667a376501a049995ab2cd8c7f8bc2253ff5bdec908: Status 404 returned error can't find the container with id e1e26c1a7f1a7bea36621667a376501a049995ab2cd8c7f8bc2253ff5bdec908
Jul 22 02:21:41.577349 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:41.577327    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k2gpv"]
Jul 22 02:21:41.747685 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:41.747659    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:41.808367 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:41.808338    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:41.896648 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:41.896613    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:42.061229 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:42.061137    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k2gpv" event={"ID":"417b06e9-0b5a-486a-92fa-aa9e118b7c36","Type":"ContainerStarted","Data":"ac7113ec341bacf79188f3d9cc638a76264dbcf682e19acd7a281d92ecba287b"}
Jul 22 02:21:42.062691 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:42.062664    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9" event={"ID":"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5","Type":"ContainerStarted","Data":"9c3ee7d1a8514bbb31d67443e5fabe9ce72639885d753f34ad57b4b432efbb46"}
Jul 22 02:21:42.062842 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:42.062706    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9" event={"ID":"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5","Type":"ContainerStarted","Data":"e1e26c1a7f1a7bea36621667a376501a049995ab2cd8c7f8bc2253ff5bdec908"}
Jul 22 02:21:42.062842 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:42.062812    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:21:42.083810 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:42.083759    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9" podStartSLOduration=47.083740244 podStartE2EDuration="47.083740244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:42.082556519 +0000 UTC m=+52.930893090" watchObservedRunningTime="2026-07-22 02:21:42.083740244 +0000 UTC m=+52.932076819"
Jul 22 02:21:42.273320 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:42.273268    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:42.472779 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:42.472741    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:42.886220 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:42.886186    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:43.484785 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:43.484737    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:43.875531 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:43.875504    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:44.068711 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:44.068678    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k2gpv" event={"ID":"417b06e9-0b5a-486a-92fa-aa9e118b7c36","Type":"ContainerStarted","Data":"6b721307c2d1c2d0848d65ecbfb0ba7cb7bec5af5d0d4400ae68729e81dc8238"}
Jul 22 02:21:44.276886 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:44.276831    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:44.401250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:44.401146    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:44.680804 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:44.680720    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:45.073917 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:45.073818    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k2gpv" event={"ID":"417b06e9-0b5a-486a-92fa-aa9e118b7c36","Type":"ContainerStarted","Data":"52b7317c3566d72addb16818521139429fa9bf2dd44e8c856cd54ef3cf148312"}
Jul 22 02:21:45.074131 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:45.073948    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:45.076751 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:45.076733    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:45.421147 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:45.421096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/50921dd8-5d4d-4801-a4c5-339f81901b9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:45.423321 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:45.423293    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/50921dd8-5d4d-4801-a4c5-339f81901b9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7984v\" (UID: \"50921dd8-5d4d-4801-a4c5-339f81901b9f\") " pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:45.472741 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:45.472709    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:45.559900 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:45.559864    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7984v"
Jul 22 02:21:45.678636 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:45.678505    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-k2gpv" podStartSLOduration=18.378348438 podStartE2EDuration="20.678478429s" totalImagesPullingTime="2.300129991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:45.096675031 +0000 UTC m=+55.945011606" watchObservedRunningTime="2026-07-22 02:21:45.678478429 +0000 UTC m=+56.526815003"
Jul 22 02:21:45.679524 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:45.679504    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7984v"]
Jul 22 02:21:45.682187 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:21:45.682144    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50921dd8_5d4d_4801_a4c5_339f81901b9f.slice/crio-9d470365b35e964b0ebc347fa9e70a5ffbcb1295c49a22d911be99b1fe42565a WatchSource:0}: Error finding container 9d470365b35e964b0ebc347fa9e70a5ffbcb1295c49a22d911be99b1fe42565a: Status 404 returned error can't find the container with id 9d470365b35e964b0ebc347fa9e70a5ffbcb1295c49a22d911be99b1fe42565a
Jul 22 02:21:45.684817 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:45.684792    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:46.073979 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:46.073891    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:46.079629 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:46.079595    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7984v" event={"ID":"50921dd8-5d4d-4801-a4c5-339f81901b9f","Type":"ContainerStarted","Data":"9d470365b35e964b0ebc347fa9e70a5ffbcb1295c49a22d911be99b1fe42565a"}
Jul 22 02:21:46.271304 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:46.271272    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:48.975139 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:48.975109    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:49.034529 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:49.034496    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:49.998157 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:49.998129    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ssvw4"
Jul 22 02:21:51.098601 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:51.098525    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7984v" event={"ID":"50921dd8-5d4d-4801-a4c5-339f81901b9f","Type":"ContainerStarted","Data":"b8a3ac3133fe6286b979e5a05cb526c95865b46966283f9facf4567f9a53f3ec"}
Jul 22 02:21:52.741989 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.741952    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-566c7f66f7-9rsq9"]
Jul 22 02:21:52.745892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.745839    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"]
Jul 22 02:21:52.753926 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.753828    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b758fbd98-nh2sj"]
Jul 22 02:21:52.756772 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.756493    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-795f5bc7b6-jhnf4"]
Jul 22 02:21:52.758378 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.758343    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"
Jul 22 02:21:52.758885 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.758340    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b758fbd98-nh2sj"
Jul 22 02:21:52.759046 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.759027    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795f5bc7b6-jhnf4"
Jul 22 02:21:52.765841 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.765689    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 22 02:21:52.765841 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.765689    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 22 02:21:52.765841 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.765750    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 22 02:21:52.766733 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.766590    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 22 02:21:52.766733 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.766612    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 22 02:21:52.766733 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.766646    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 22 02:21:52.766733 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.766656    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 22 02:21:52.767366 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.767341    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 22 02:21:52.768173 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.768152    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 22 02:21:52.773526 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.773505    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-9bmhx\""
Jul 22 02:21:52.773991 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.773682    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-795f5bc7b6-jhnf4"]
Jul 22 02:21:52.774946 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.774917    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"]
Jul 22 02:21:52.775698 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.775677    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b758fbd98-nh2sj"]
Jul 22 02:21:52.882658 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.882621    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/76e048bf-a787-400c-8811-4ab546c101a5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-855599d895-qstkp\" (UID: \"76e048bf-a787-400c-8811-4ab546c101a5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"
Jul 22 02:21:52.882866 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.882672    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/66a98cfb-a8ef-4db0-8637-b27c4fe11b74-klusterlet-config\") pod \"klusterlet-addon-workmgr-795f5bc7b6-jhnf4\" (UID: \"66a98cfb-a8ef-4db0-8637-b27c4fe11b74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795f5bc7b6-jhnf4"
Jul 22 02:21:52.882866 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.882725    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/76e048bf-a787-400c-8811-4ab546c101a5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-855599d895-qstkp\" (UID: \"76e048bf-a787-400c-8811-4ab546c101a5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"
Jul 22 02:21:52.882866 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.882832    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66a98cfb-a8ef-4db0-8637-b27c4fe11b74-tmp\") pod \"klusterlet-addon-workmgr-795f5bc7b6-jhnf4\" (UID: \"66a98cfb-a8ef-4db0-8637-b27c4fe11b74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795f5bc7b6-jhnf4"
Jul 22 02:21:52.883016 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.882921    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/76e048bf-a787-400c-8811-4ab546c101a5-ca\") pod \"cluster-proxy-proxy-agent-855599d895-qstkp\" (UID: \"76e048bf-a787-400c-8811-4ab546c101a5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"
Jul 22 02:21:52.883016 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.882962    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92162137-026d-4f4b-b6c1-fc606dce2515-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b758fbd98-nh2sj\" (UID: \"92162137-026d-4f4b-b6c1-fc606dce2515\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b758fbd98-nh2sj"
Jul 22 02:21:52.883106 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.883010    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qs2k\" (UniqueName: \"kubernetes.io/projected/66a98cfb-a8ef-4db0-8637-b27c4fe11b74-kube-api-access-2qs2k\") pod \"klusterlet-addon-workmgr-795f5bc7b6-jhnf4\" (UID: \"66a98cfb-a8ef-4db0-8637-b27c4fe11b74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795f5bc7b6-jhnf4"
Jul 22 02:21:52.883106 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.883053    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp7th\" (UniqueName: \"kubernetes.io/projected/76e048bf-a787-400c-8811-4ab546c101a5-kube-api-access-pp7th\") pod \"cluster-proxy-proxy-agent-855599d895-qstkp\" (UID: \"76e048bf-a787-400c-8811-4ab546c101a5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"
Jul 22 02:21:52.883106 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.883084    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvmm8\" (UniqueName: \"kubernetes.io/projected/92162137-026d-4f4b-b6c1-fc606dce2515-kube-api-access-dvmm8\") pod \"managed-serviceaccount-addon-agent-5b758fbd98-nh2sj\" (UID: \"92162137-026d-4f4b-b6c1-fc606dce2515\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b758fbd98-nh2sj"
Jul 22 02:21:52.883249 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.883133    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76e048bf-a787-400c-8811-4ab546c101a5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-855599d895-qstkp\" (UID: \"76e048bf-a787-400c-8811-4ab546c101a5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"
Jul 22 02:21:52.883249 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.883173    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/76e048bf-a787-400c-8811-4ab546c101a5-hub\") pod \"cluster-proxy-proxy-agent-855599d895-qstkp\" (UID: \"76e048bf-a787-400c-8811-4ab546c101a5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"
Jul 22 02:21:52.886657 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.886632    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cff98f456-jkdf8"]
Jul 22 02:21:52.889555 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.889535    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:52.912328 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.912300    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cff98f456-jkdf8"]
Jul 22 02:21:52.941016 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.940980    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:52.984109 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.984079    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e6149cf-f721-48fa-9255-04dcb0486502-ca-trust-extracted\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:52.984247 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.984137    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66a98cfb-a8ef-4db0-8637-b27c4fe11b74-tmp\") pod \"klusterlet-addon-workmgr-795f5bc7b6-jhnf4\" (UID: \"66a98cfb-a8ef-4db0-8637-b27c4fe11b74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795f5bc7b6-jhnf4"
Jul 22 02:21:52.984247 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.984162    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/76e048bf-a787-400c-8811-4ab546c101a5-ca\") pod \"cluster-proxy-proxy-agent-855599d895-qstkp\" (UID: \"76e048bf-a787-400c-8811-4ab546c101a5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"
Jul 22 02:21:52.984247 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.984178    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92162137-026d-4f4b-b6c1-fc606dce2515-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b758fbd98-nh2sj\" (UID: \"92162137-026d-4f4b-b6c1-fc606dce2515\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b758fbd98-nh2sj"
Jul 22 02:21:52.984247 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.984203    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjlnk\" (UniqueName: \"kubernetes.io/projected/1e6149cf-f721-48fa-9255-04dcb0486502-kube-api-access-xjlnk\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:52.984460 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.984320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qs2k\" (UniqueName: \"kubernetes.io/projected/66a98cfb-a8ef-4db0-8637-b27c4fe11b74-kube-api-access-2qs2k\") pod \"klusterlet-addon-workmgr-795f5bc7b6-jhnf4\" (UID: \"66a98cfb-a8ef-4db0-8637-b27c4fe11b74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795f5bc7b6-jhnf4"
Jul 22 02:21:52.984460 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.984430    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp7th\" (UniqueName: \"kubernetes.io/projected/76e048bf-a787-400c-8811-4ab546c101a5-kube-api-access-pp7th\") pod \"cluster-proxy-proxy-agent-855599d895-qstkp\" (UID: \"76e048bf-a787-400c-8811-4ab546c101a5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"
Jul 22 02:21:52.984561 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.984471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvmm8\" (UniqueName: \"kubernetes.io/projected/92162137-026d-4f4b-b6c1-fc606dce2515-kube-api-access-dvmm8\") pod \"managed-serviceaccount-addon-agent-5b758fbd98-nh2sj\" (UID: \"92162137-026d-4f4b-b6c1-fc606dce2515\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b758fbd98-nh2sj"
Jul 22 02:21:52.984645 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.984615    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e6149cf-f721-48fa-9255-04dcb0486502-installation-pull-secrets\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:52.984719 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.984682    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76e048bf-a787-400c-8811-4ab546c101a5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-855599d895-qstkp\" (UID: \"76e048bf-a787-400c-8811-4ab546c101a5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"
Jul 22 02:21:52.984768 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.984718    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e6149cf-f721-48fa-9255-04dcb0486502-bound-sa-token\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:52.984768 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.984735    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66a98cfb-a8ef-4db0-8637-b27c4fe11b74-tmp\") pod \"klusterlet-addon-workmgr-795f5bc7b6-jhnf4\" (UID: \"66a98cfb-a8ef-4db0-8637-b27c4fe11b74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795f5bc7b6-jhnf4"
Jul 22 02:21:52.984915 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.984765    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e6149cf-f721-48fa-9255-04dcb0486502-image-registry-private-configuration\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:52.984915 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.984788    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e6149cf-f721-48fa-9255-04dcb0486502-registry-tls\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:52.984915 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.984814    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/76e048bf-a787-400c-8811-4ab546c101a5-hub\") pod \"cluster-proxy-proxy-agent-855599d895-qstkp\" (UID: \"76e048bf-a787-400c-8811-4ab546c101a5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"
Jul 22 02:21:52.984915 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.984882    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e6149cf-f721-48fa-9255-04dcb0486502-trusted-ca\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:52.985136 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.984931    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e6149cf-f721-48fa-9255-04dcb0486502-registry-certificates\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:52.985136 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.984967    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/76e048bf-a787-400c-8811-4ab546c101a5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-855599d895-qstkp\" (UID: \"76e048bf-a787-400c-8811-4ab546c101a5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"
Jul 22 02:21:52.985136 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.985001    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/66a98cfb-a8ef-4db0-8637-b27c4fe11b74-klusterlet-config\") pod \"klusterlet-addon-workmgr-795f5bc7b6-jhnf4\" (UID: \"66a98cfb-a8ef-4db0-8637-b27c4fe11b74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795f5bc7b6-jhnf4"
Jul 22 02:21:52.985136 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.985045    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/76e048bf-a787-400c-8811-4ab546c101a5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-855599d895-qstkp\" (UID: \"76e048bf-a787-400c-8811-4ab546c101a5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"
Jul 22 02:21:52.985678 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.985658    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/76e048bf-a787-400c-8811-4ab546c101a5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-855599d895-qstkp\" (UID: \"76e048bf-a787-400c-8811-4ab546c101a5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"
Jul 22 02:21:52.987430 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.987407    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/76e048bf-a787-400c-8811-4ab546c101a5-ca\") pod \"cluster-proxy-proxy-agent-855599d895-qstkp\" (UID: \"76e048bf-a787-400c-8811-4ab546c101a5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"
Jul 22 02:21:52.987631 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.987584    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92162137-026d-4f4b-b6c1-fc606dce2515-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b758fbd98-nh2sj\" (UID: \"92162137-026d-4f4b-b6c1-fc606dce2515\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b758fbd98-nh2sj"
Jul 22 02:21:52.987865 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.987805    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/66a98cfb-a8ef-4db0-8637-b27c4fe11b74-klusterlet-config\") pod \"klusterlet-addon-workmgr-795f5bc7b6-jhnf4\" (UID: \"66a98cfb-a8ef-4db0-8637-b27c4fe11b74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795f5bc7b6-jhnf4"
Jul 22 02:21:52.988050 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.988031    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/76e048bf-a787-400c-8811-4ab546c101a5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-855599d895-qstkp\" (UID: \"76e048bf-a787-400c-8811-4ab546c101a5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"
Jul 22 02:21:52.988250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.988228    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76e048bf-a787-400c-8811-4ab546c101a5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-855599d895-qstkp\" (UID: \"76e048bf-a787-400c-8811-4ab546c101a5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"
Jul 22 02:21:52.988578 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.988560    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/76e048bf-a787-400c-8811-4ab546c101a5-hub\") pod \"cluster-proxy-proxy-agent-855599d895-qstkp\" (UID: \"76e048bf-a787-400c-8811-4ab546c101a5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"
Jul 22 02:21:52.994385 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.994240    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp7th\" (UniqueName: \"kubernetes.io/projected/76e048bf-a787-400c-8811-4ab546c101a5-kube-api-access-pp7th\") pod \"cluster-proxy-proxy-agent-855599d895-qstkp\" (UID: \"76e048bf-a787-400c-8811-4ab546c101a5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"
Jul 22 02:21:52.994494 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.994475    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qs2k\" (UniqueName: \"kubernetes.io/projected/66a98cfb-a8ef-4db0-8637-b27c4fe11b74-kube-api-access-2qs2k\") pod \"klusterlet-addon-workmgr-795f5bc7b6-jhnf4\" (UID: \"66a98cfb-a8ef-4db0-8637-b27c4fe11b74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795f5bc7b6-jhnf4"
Jul 22 02:21:52.994704 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:52.994685    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvmm8\" (UniqueName: \"kubernetes.io/projected/92162137-026d-4f4b-b6c1-fc606dce2515-kube-api-access-dvmm8\") pod \"managed-serviceaccount-addon-agent-5b758fbd98-nh2sj\" (UID: \"92162137-026d-4f4b-b6c1-fc606dce2515\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b758fbd98-nh2sj"
Jul 22 02:21:53.036083 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.036039    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:53.086326 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.086296    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e6149cf-f721-48fa-9255-04dcb0486502-trusted-ca\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:53.086471 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.086423    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e6149cf-f721-48fa-9255-04dcb0486502-registry-certificates\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:53.086549 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.086470    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e6149cf-f721-48fa-9255-04dcb0486502-ca-trust-extracted\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:53.086549 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.086520    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjlnk\" (UniqueName: \"kubernetes.io/projected/1e6149cf-f721-48fa-9255-04dcb0486502-kube-api-access-xjlnk\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:53.086663 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.086570    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e6149cf-f721-48fa-9255-04dcb0486502-installation-pull-secrets\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:53.086663 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.086609    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e6149cf-f721-48fa-9255-04dcb0486502-bound-sa-token\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:53.086663 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.086646    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e6149cf-f721-48fa-9255-04dcb0486502-image-registry-private-configuration\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:53.086811 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.086690    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e6149cf-f721-48fa-9255-04dcb0486502-registry-tls\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:53.087288 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.087244    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e6149cf-f721-48fa-9255-04dcb0486502-registry-certificates\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:53.087288 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.087273    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e6149cf-f721-48fa-9255-04dcb0486502-trusted-ca\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:53.087468 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.087407    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e6149cf-f721-48fa-9255-04dcb0486502-ca-trust-extracted\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:53.089622 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.089590    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e6149cf-f721-48fa-9255-04dcb0486502-image-registry-private-configuration\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:53.089622 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.089603    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e6149cf-f721-48fa-9255-04dcb0486502-registry-tls\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:53.089813 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.089709    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e6149cf-f721-48fa-9255-04dcb0486502-installation-pull-secrets\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:53.089813 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.089731    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"
Jul 22 02:21:53.096124 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.096086    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b758fbd98-nh2sj"
Jul 22 02:21:53.099592 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.099574    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e6149cf-f721-48fa-9255-04dcb0486502-bound-sa-token\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:53.099681 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.099625    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjlnk\" (UniqueName: \"kubernetes.io/projected/1e6149cf-f721-48fa-9255-04dcb0486502-kube-api-access-xjlnk\") pod \"image-registry-7cff98f456-jkdf8\" (UID: \"1e6149cf-f721-48fa-9255-04dcb0486502\") " pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:53.112773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.112753    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795f5bc7b6-jhnf4"
Jul 22 02:21:53.136772 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.136730    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:53.199133 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.199107    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:53.235532 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.235482    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:53.596316 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.596285    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:53.671285 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.671244    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:53.735126 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.735099    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:53.837508 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.837478    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:53.947155 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:53.947121    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:54.322323 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:54.322247    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:54.546699 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:54.546682    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp"]
Jul 22 02:21:54.549293 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:21:54.549267    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76e048bf_a787_400c_8811_4ab546c101a5.slice/crio-796f30ea2018f3e8188f5026241921705a302dd662bf3d4893c6456454b67485 WatchSource:0}: Error finding container 796f30ea2018f3e8188f5026241921705a302dd662bf3d4893c6456454b67485: Status 404 returned error can't find the container with id 796f30ea2018f3e8188f5026241921705a302dd662bf3d4893c6456454b67485
Jul 22 02:21:54.747460 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:54.747425    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:54.769917 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:54.769889    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b758fbd98-nh2sj"]
Jul 22 02:21:54.772799 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:21:54.772773    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92162137_026d_4f4b_b6c1_fc606dce2515.slice/crio-40ad8943934349870772f576c0516614a89ddde0cc902bfd6d33eb917755b5e4 WatchSource:0}: Error finding container 40ad8943934349870772f576c0516614a89ddde0cc902bfd6d33eb917755b5e4: Status 404 returned error can't find the container with id 40ad8943934349870772f576c0516614a89ddde0cc902bfd6d33eb917755b5e4
Jul 22 02:21:54.772925 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:54.772911    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-795f5bc7b6-jhnf4"]
Jul 22 02:21:54.776012 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:21:54.775987    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a98cfb_a8ef_4db0_8637_b27c4fe11b74.slice/crio-5aadb1c761c0dd8d0ffc52b5cb77bd2c4f9c34716df98a53505190c2f7519807 WatchSource:0}: Error finding container 5aadb1c761c0dd8d0ffc52b5cb77bd2c4f9c34716df98a53505190c2f7519807: Status 404 returned error can't find the container with id 5aadb1c761c0dd8d0ffc52b5cb77bd2c4f9c34716df98a53505190c2f7519807
Jul 22 02:21:54.776573 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:54.776537    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cff98f456-jkdf8"]
Jul 22 02:21:54.779090 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:21:54.779067    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6149cf_f721_48fa_9255_04dcb0486502.slice/crio-33450e335449e05f81e268363f3f5c0e6a3d466628a8bfd390c76b9c2fc362f2 WatchSource:0}: Error finding container 33450e335449e05f81e268363f3f5c0e6a3d466628a8bfd390c76b9c2fc362f2: Status 404 returned error can't find the container with id 33450e335449e05f81e268363f3f5c0e6a3d466628a8bfd390c76b9c2fc362f2
Jul 22 02:21:55.082638 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.082550    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k2gpv"
Jul 22 02:21:55.116095 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.115562    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cff98f456-jkdf8" event={"ID":"1e6149cf-f721-48fa-9255-04dcb0486502","Type":"ContainerStarted","Data":"f14ed16a3be68f6c4fab54f19e87284e2b7e437bfcbec6c243a7c4f9fc455d3b"}
Jul 22 02:21:55.116095 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.115626    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cff98f456-jkdf8" event={"ID":"1e6149cf-f721-48fa-9255-04dcb0486502","Type":"ContainerStarted","Data":"33450e335449e05f81e268363f3f5c0e6a3d466628a8bfd390c76b9c2fc362f2"}
Jul 22 02:21:55.116095 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.115664    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:21:55.117167 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.117137    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795f5bc7b6-jhnf4" event={"ID":"66a98cfb-a8ef-4db0-8637-b27c4fe11b74","Type":"ContainerStarted","Data":"5aadb1c761c0dd8d0ffc52b5cb77bd2c4f9c34716df98a53505190c2f7519807"}
Jul 22 02:21:55.118486 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.118452    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b758fbd98-nh2sj" event={"ID":"92162137-026d-4f4b-b6c1-fc606dce2515","Type":"ContainerStarted","Data":"40ad8943934349870772f576c0516614a89ddde0cc902bfd6d33eb917755b5e4"}
Jul 22 02:21:55.119885 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.119815    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp" event={"ID":"76e048bf-a787-400c-8811-4ab546c101a5","Type":"ContainerStarted","Data":"796f30ea2018f3e8188f5026241921705a302dd662bf3d4893c6456454b67485"}
Jul 22 02:21:55.123097 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.123076    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7984v" event={"ID":"50921dd8-5d4d-4801-a4c5-339f81901b9f","Type":"ContainerStarted","Data":"63f6b366f9a7afb0e6b6beb67b2de13e5f5499cc3d197a0e508f62fcf2055dae"}
Jul 22 02:21:55.134485 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.134442    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7cff98f456-jkdf8" podStartSLOduration=3.134429769 podStartE2EDuration="3.134429769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:55.133804884 +0000 UTC m=+65.982141467" watchObservedRunningTime="2026-07-22 02:21:55.134429769 +0000 UTC m=+65.982766321"
Jul 22 02:21:55.155113 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.155085    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:55.256611 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.256575    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:55.357375 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.357341    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:55.407764 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.407674    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret\") pod \"global-pull-secret-syncer-kzv8x\" (UID: \"faebb5f3-7db6-4e1c-88b0-a4530442e728\") " pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:21:55.410646 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.410437    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 22 02:21:55.421110 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.421083    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/faebb5f3-7db6-4e1c-88b0-a4530442e728-original-pull-secret\") pod \"global-pull-secret-syncer-kzv8x\" (UID: \"faebb5f3-7db6-4e1c-88b0-a4530442e728\") " pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:21:55.509530 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.509130    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs\") pod \"network-metrics-daemon-p8jzv\" (UID: \"ba53539c-579d-43a1-a6d0-d9020f0d18f3\") " pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:21:55.511652 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.511624    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 22 02:21:55.524949 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.524920    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba53539c-579d-43a1-a6d0-d9020f0d18f3-metrics-certs\") pod \"network-metrics-daemon-p8jzv\" (UID: \"ba53539c-579d-43a1-a6d0-d9020f0d18f3\") " pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:21:55.531924 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.531881    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kzv8x"
Jul 22 02:21:55.548254 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.548063    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8mll8\""
Jul 22 02:21:55.556307 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.556106    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8jzv"
Jul 22 02:21:55.610942 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.610330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmgc\" (UniqueName: \"kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc\") pod \"network-check-target-r4hdg\" (UID: \"0036981e-4941-42f3-91d1-d1c98ffa1f5e\") " pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:21:55.613275 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.613064    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 22 02:21:55.623686 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.623481    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 22 02:21:55.638112 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.638048    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdmgc\" (UniqueName: \"kubernetes.io/projected/0036981e-4941-42f3-91d1-d1c98ffa1f5e-kube-api-access-sdmgc\") pod \"network-check-target-r4hdg\" (UID: \"0036981e-4941-42f3-91d1-d1c98ffa1f5e\") " pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:21:55.791710 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.791677    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kzv8x"]
Jul 22 02:21:55.804095 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:21:55.803965    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaebb5f3_7db6_4e1c_88b0_a4530442e728.slice/crio-2e1df64ee64c7d85aab0d0745c265dc0ea75c2555059d2779d87bf69df0b97ba WatchSource:0}: Error finding container 2e1df64ee64c7d85aab0d0745c265dc0ea75c2555059d2779d87bf69df0b97ba: Status 404 returned error can't find the container with id 2e1df64ee64c7d85aab0d0745c265dc0ea75c2555059d2779d87bf69df0b97ba
Jul 22 02:21:55.804457 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.804388    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p8jzv"]
Jul 22 02:21:55.807945 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:21:55.807906    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba53539c_579d_43a1_a6d0_d9020f0d18f3.slice/crio-23d136848ae877ab75993a6c993199daf84608e4af8cc58f5da7b55b018c6404 WatchSource:0}: Error finding container 23d136848ae877ab75993a6c993199daf84608e4af8cc58f5da7b55b018c6404: Status 404 returned error can't find the container with id 23d136848ae877ab75993a6c993199daf84608e4af8cc58f5da7b55b018c6404
Jul 22 02:21:55.842314 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.842293    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cw2sm\""
Jul 22 02:21:55.850459 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:55.850439    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:21:56.028823 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:56.028729    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r4hdg"]
Jul 22 02:21:56.033638 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:21:56.033607    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0036981e_4941_42f3_91d1_d1c98ffa1f5e.slice/crio-1f30778a75db9c57a03a1400e45054abd381755dc92a000b2e84edbf27596e01 WatchSource:0}: Error finding container 1f30778a75db9c57a03a1400e45054abd381755dc92a000b2e84edbf27596e01: Status 404 returned error can't find the container with id 1f30778a75db9c57a03a1400e45054abd381755dc92a000b2e84edbf27596e01
Jul 22 02:21:56.133441 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:56.133291    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r4hdg" event={"ID":"0036981e-4941-42f3-91d1-d1c98ffa1f5e","Type":"ContainerStarted","Data":"1f30778a75db9c57a03a1400e45054abd381755dc92a000b2e84edbf27596e01"}
Jul 22 02:21:56.136578 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:56.136475    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8jzv" event={"ID":"ba53539c-579d-43a1-a6d0-d9020f0d18f3","Type":"ContainerStarted","Data":"23d136848ae877ab75993a6c993199daf84608e4af8cc58f5da7b55b018c6404"}
Jul 22 02:21:56.138091 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:56.138029    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kzv8x" event={"ID":"faebb5f3-7db6-4e1c-88b0-a4530442e728","Type":"ContainerStarted","Data":"2e1df64ee64c7d85aab0d0745c265dc0ea75c2555059d2779d87bf69df0b97ba"}
Jul 22 02:21:56.246722 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:56.246628    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:57.575627 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:57.575593    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:57.631222 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:57.631152    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:59.575712 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:59.575665    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:59.676074 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:59.676038    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:59.758628 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:59.758587    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:21:59.965942 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:21:59.965871    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:01.571081 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:01.571046    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:01.665042 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:01.664952    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:02.611151 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:02.611109    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:02.688502 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:02.688472    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:02.748475 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:02.748451    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:22:03.761296 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:03.761255    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:03.861168 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:03.861130    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:03.962350 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:03.962322    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:04.042281 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:04.042189    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:04.107751 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:04.107712    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:04.221509 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:04.221474    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:04.277356 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:04.277321    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:05.403918 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:05.403877    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:05.497042 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:05.497000    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:07.190541 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.190503    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp" event={"ID":"76e048bf-a787-400c-8811-4ab546c101a5","Type":"ContainerStarted","Data":"75ac98582cdf561ab681e0f3c6574452991272297f55c1630e8655c173b53711"}
Jul 22 02:22:07.191001 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.190547    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp" event={"ID":"76e048bf-a787-400c-8811-4ab546c101a5","Type":"ContainerStarted","Data":"5cbbed2360d916d39ffd6786b87bafa209fcb81032550295c05db9fcf0fb0866"}
Jul 22 02:22:07.191001 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.190565    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp" event={"ID":"76e048bf-a787-400c-8811-4ab546c101a5","Type":"ContainerStarted","Data":"6418794d0135268ec3e752a93eb565514ab30ed3b88daaa9797b3ae63521442e"}
Jul 22 02:22:07.192734 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.192694    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r4hdg" event={"ID":"0036981e-4941-42f3-91d1-d1c98ffa1f5e","Type":"ContainerStarted","Data":"42ff5036e6a308d0b5b9e9802e4bbc656e7aec9edc8d52bc8161343d23ceaf82"}
Jul 22 02:22:07.192958 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.192788    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:22:07.194921 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.194891    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7984v" event={"ID":"50921dd8-5d4d-4801-a4c5-339f81901b9f","Type":"ContainerStarted","Data":"64d0c9ed6da7e8fa2d8d5f50345558e798a409840dbe35ff196d25ea34798248"}
Jul 22 02:22:07.196656 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.196635    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8jzv" event={"ID":"ba53539c-579d-43a1-a6d0-d9020f0d18f3","Type":"ContainerStarted","Data":"da96c2cd3cd50520924e7bd3c3448419136e14eab8038946c6a8d4127a0454f5"}
Jul 22 02:22:07.196748 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.196673    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8jzv" event={"ID":"ba53539c-579d-43a1-a6d0-d9020f0d18f3","Type":"ContainerStarted","Data":"f7ba85ab86948b9266acefa202cfc3b9cb7732a57f3e44ae4569422af48d69bb"}
Jul 22 02:22:07.198041 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.198017    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kzv8x" event={"ID":"faebb5f3-7db6-4e1c-88b0-a4530442e728","Type":"ContainerStarted","Data":"e31628aaa48aa42b80f4bda63f7e6a53ad3821fcf621d9d73ed0589e6ca558e8"}
Jul 22 02:22:07.199359 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.199339    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795f5bc7b6-jhnf4" event={"ID":"66a98cfb-a8ef-4db0-8637-b27c4fe11b74","Type":"ContainerStarted","Data":"59004e1ec5cb5d7eca215068c6d5a1010466a32ea1e0f5a45553f21f5ff1a2d7"}
Jul 22 02:22:07.199481 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.199453    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795f5bc7b6-jhnf4"
Jul 22 02:22:07.200754 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.200732    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b758fbd98-nh2sj" event={"ID":"92162137-026d-4f4b-b6c1-fc606dce2515","Type":"ContainerStarted","Data":"23b4a2436dee0344757691f23c13fad8292c072ba5b8972c8a0a51b4f23e6e91"}
Jul 22 02:22:07.201475 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.201459    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795f5bc7b6-jhnf4"
Jul 22 02:22:07.209922 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.209876    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-855599d895-qstkp" podStartSLOduration=3.097854304 podStartE2EDuration="15.209837757s" totalImagesPullingTime="12.111983453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:22:07.208436535 +0000 UTC m=+78.056773126" watchObservedRunningTime="2026-07-22 02:22:07.209837757 +0000 UTC m=+78.058174330"
Jul 22 02:22:07.223440 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.223404    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-r4hdg" podStartSLOduration=67.617250959 podStartE2EDuration="1m18.223390976s" totalImagesPullingTime="10.606140017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:22:07.222492345 +0000 UTC m=+78.070828920" watchObservedRunningTime="2026-07-22 02:22:07.223390976 +0000 UTC m=+78.071727528"
Jul 22 02:22:07.238822 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.238780    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b758fbd98-nh2sj" podStartSLOduration=3.37350169 podStartE2EDuration="15.238764681s" totalImagesPullingTime="11.865262991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:22:07.238366949 +0000 UTC m=+78.086703536" watchObservedRunningTime="2026-07-22 02:22:07.238764681 +0000 UTC m=+78.087101259"
Jul 22 02:22:07.253648 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.253606    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p8jzv" podStartSLOduration=67.471890607 podStartE2EDuration="1m18.253592015s" totalImagesPullingTime="10.781701408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:22:07.253365227 +0000 UTC m=+78.101701802" watchObservedRunningTime="2026-07-22 02:22:07.253592015 +0000 UTC m=+78.101928590"
Jul 22 02:22:07.277136 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.277096    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7984v" podStartSLOduration=17.538768222 podStartE2EDuration="38.277085342s" totalImagesPullingTime="20.73831712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:21:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:22:07.276487374 +0000 UTC m=+78.124823950" watchObservedRunningTime="2026-07-22 02:22:07.277085342 +0000 UTC m=+78.125421946"
Jul 22 02:22:07.294319 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.294283    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-795f5bc7b6-jhnf4" podStartSLOduration=3.430329197 podStartE2EDuration="15.29426953s" totalImagesPullingTime="11.863940333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:22:07.29404456 +0000 UTC m=+78.142381147" watchObservedRunningTime="2026-07-22 02:22:07.29426953 +0000 UTC m=+78.142606106"
Jul 22 02:22:07.314435 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.314400    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kzv8x" podStartSLOduration=67.454486421 podStartE2EDuration="1m18.314391629s" totalImagesPullingTime="10.859905208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:22:07.313999401 +0000 UTC m=+78.162335975" watchObservedRunningTime="2026-07-22 02:22:07.314391629 +0000 UTC m=+78.162728203"
Jul 22 02:22:07.372053 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.372031    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:07.480743 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:07.480672    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:08.804031 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:08.803996    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:08.901673 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:08.901642    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:11.005311 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:11.005277    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:11.105593 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:11.105554    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:11.207094 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:11.207062    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:11.714501 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:11.714404    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:11.815612 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:11.815582    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:12.585568 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:12.585537    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:12.648042 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:12.648008    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:14.550731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:14.550700    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:14.650941 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:14.650916    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:16.143003 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:16.142975    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7cff98f456-jkdf8"
Jul 22 02:22:16.268992 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:16.268963    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:16.322961 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:16.322936    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:16.550315 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:16.550244    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:16.651459 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:16.651434    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:17.766834 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:17.766799    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9" podUID="9b9cc81f-d487-41fd-b3f4-8ae35600b8c5" containerName="registry" containerID="cri-o://9c3ee7d1a8514bbb31d67443e5fabe9ce72639885d753f34ad57b4b432efbb46" gracePeriod=30
Jul 22 02:22:17.901142 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:17.901116    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:18.082718 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.082693    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:18.182732 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.182698    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:18.238580 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.238554    2565 generic.go:356] "Generic (PLEG): container finished" podID="9b9cc81f-d487-41fd-b3f4-8ae35600b8c5" containerID="9c3ee7d1a8514bbb31d67443e5fabe9ce72639885d753f34ad57b4b432efbb46" exitCode=0
Jul 22 02:22:18.238689 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.238598    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9" event={"ID":"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5","Type":"ContainerDied","Data":"9c3ee7d1a8514bbb31d67443e5fabe9ce72639885d753f34ad57b4b432efbb46"}
Jul 22 02:22:18.552096 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.552070    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:22:18.700336 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.700260    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-installation-pull-secrets\") pod \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") "
Jul 22 02:22:18.700336 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.700318    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-ca-trust-extracted\") pod \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") "
Jul 22 02:22:18.700554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.700348    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9l4x2\" (UniqueName: \"kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-kube-api-access-9l4x2\") pod \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") "
Jul 22 02:22:18.700554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.700381    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-certificates\") pod \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") "
Jul 22 02:22:18.700554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.700407    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-bound-sa-token\") pod \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") "
Jul 22 02:22:18.700554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.700436    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-image-registry-private-configuration\") pod \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") "
Jul 22 02:22:18.700554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.700464    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-trusted-ca\") pod \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") "
Jul 22 02:22:18.700554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.700501    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls\") pod \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\" (UID: \"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5\") "
Jul 22 02:22:18.701334 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.700932    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5" (UID: "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:22:18.701540 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.701411    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5" (UID: "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:22:18.703176 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.703145    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5" (UID: "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:22:18.703353 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.703304    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5" (UID: "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:22:18.703465 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.703398    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5" (UID: "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:22:18.703465 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.703432    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5" (UID: "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:22:18.703580 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.703498    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-kube-api-access-9l4x2" (OuterVolumeSpecName: "kube-api-access-9l4x2") pod "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5" (UID: "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5"). InnerVolumeSpecName "kube-api-access-9l4x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:22:18.710536 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.710515    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5" (UID: "9b9cc81f-d487-41fd-b3f4-8ae35600b8c5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:22:18.802057 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.802025    2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-ca-trust-extracted\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:22:18.802057 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.802050    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l4x2\" (UniqueName: \"kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-kube-api-access-9l4x2\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:22:18.802057 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.802059    2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-certificates\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:22:18.802451 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.802068    2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-bound-sa-token\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:22:18.802451 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.802078    2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-image-registry-private-configuration\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:22:18.802451 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.802089    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-trusted-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:22:18.802451 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.802097    2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-registry-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:22:18.802451 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:18.802104    2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5-installation-pull-secrets\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:22:19.243772 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:19.243742    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9"
Jul 22 02:22:19.243966 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:19.243742    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-566c7f66f7-9rsq9" event={"ID":"9b9cc81f-d487-41fd-b3f4-8ae35600b8c5","Type":"ContainerDied","Data":"e1e26c1a7f1a7bea36621667a376501a049995ab2cd8c7f8bc2253ff5bdec908"}
Jul 22 02:22:19.243966 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:19.243905    2565 scope.go:117] "RemoveContainer" containerID="9c3ee7d1a8514bbb31d67443e5fabe9ce72639885d753f34ad57b4b432efbb46"
Jul 22 02:22:19.263687 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:19.263660    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-566c7f66f7-9rsq9"]
Jul 22 02:22:19.269879 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:19.269839    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-566c7f66f7-9rsq9"]
Jul 22 02:22:19.559198 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:19.559115    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:19.611621 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:19.611592    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:19.724546 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:19.724511    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b9cc81f-d487-41fd-b3f4-8ae35600b8c5" path="/var/lib/kubelet/pods/9b9cc81f-d487-41fd-b3f4-8ae35600b8c5/volumes"
Jul 22 02:22:19.892033 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:19.891985    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:19.991835 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:19.991789    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:20.104052 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.104021    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:20.176335 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.176261    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t74nq"]
Jul 22 02:22:20.176596 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.176583    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b9cc81f-d487-41fd-b3f4-8ae35600b8c5" containerName="registry"
Jul 22 02:22:20.176639 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.176599    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b9cc81f-d487-41fd-b3f4-8ae35600b8c5" containerName="registry"
Jul 22 02:22:20.176685 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.176662    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b9cc81f-d487-41fd-b3f4-8ae35600b8c5" containerName="registry"
Jul 22 02:22:20.199203 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.199171    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:20.251396 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.251372    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.253699 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.253678    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 22 02:22:20.253835 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.253677    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 22 02:22:20.253835 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.253798    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 22 02:22:20.254573 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.254552    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 22 02:22:20.254743 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.254724    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 22 02:22:20.254743 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.254733    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sjx6g\""
Jul 22 02:22:20.254895 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.254757    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 22 02:22:20.265410 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.265379    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:20.414868 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.414819    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9dd103b8-b2d4-45fc-9194-9de212c9410f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.415015 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.414875    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9dd103b8-b2d4-45fc-9194-9de212c9410f-node-exporter-wtmp\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.415015 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.414901    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9dd103b8-b2d4-45fc-9194-9de212c9410f-node-exporter-tls\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.415015 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.414972    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc4dl\" (UniqueName: \"kubernetes.io/projected/9dd103b8-b2d4-45fc-9194-9de212c9410f-kube-api-access-tc4dl\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.415015 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.414996    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9dd103b8-b2d4-45fc-9194-9de212c9410f-sys\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.415015 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.415010    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9dd103b8-b2d4-45fc-9194-9de212c9410f-root\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.415167 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.415032    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dd103b8-b2d4-45fc-9194-9de212c9410f-metrics-client-ca\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.415167 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.415086    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9dd103b8-b2d4-45fc-9194-9de212c9410f-node-exporter-textfile\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.415167 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.415111    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9dd103b8-b2d4-45fc-9194-9de212c9410f-node-exporter-accelerators-collector-config\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.430502 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.430435    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:20.516280 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.516255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9dd103b8-b2d4-45fc-9194-9de212c9410f-node-exporter-accelerators-collector-config\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.516442 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.516287    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9dd103b8-b2d4-45fc-9194-9de212c9410f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.516442 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.516307    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9dd103b8-b2d4-45fc-9194-9de212c9410f-node-exporter-wtmp\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.516442 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.516335    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9dd103b8-b2d4-45fc-9194-9de212c9410f-node-exporter-tls\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.516442 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.516364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc4dl\" (UniqueName: \"kubernetes.io/projected/9dd103b8-b2d4-45fc-9194-9de212c9410f-kube-api-access-tc4dl\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.516442 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.516380    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9dd103b8-b2d4-45fc-9194-9de212c9410f-sys\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.516442 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.516396    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9dd103b8-b2d4-45fc-9194-9de212c9410f-root\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.516442 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.516426    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dd103b8-b2d4-45fc-9194-9de212c9410f-metrics-client-ca\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.516809 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.516462    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9dd103b8-b2d4-45fc-9194-9de212c9410f-node-exporter-textfile\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.516809 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.516471    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9dd103b8-b2d4-45fc-9194-9de212c9410f-node-exporter-wtmp\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.516809 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.516485    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9dd103b8-b2d4-45fc-9194-9de212c9410f-root\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.516809 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.516491    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9dd103b8-b2d4-45fc-9194-9de212c9410f-sys\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.516809 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.516785    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9dd103b8-b2d4-45fc-9194-9de212c9410f-node-exporter-textfile\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.517054 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.516963    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9dd103b8-b2d4-45fc-9194-9de212c9410f-node-exporter-accelerators-collector-config\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.517054 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.516975    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dd103b8-b2d4-45fc-9194-9de212c9410f-metrics-client-ca\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.518645 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.518625    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9dd103b8-b2d4-45fc-9194-9de212c9410f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.518833 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.518790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9dd103b8-b2d4-45fc-9194-9de212c9410f-node-exporter-tls\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.524127 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.524108    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc4dl\" (UniqueName: \"kubernetes.io/projected/9dd103b8-b2d4-45fc-9194-9de212c9410f-kube-api-access-tc4dl\") pod \"node-exporter-t74nq\" (UID: \"9dd103b8-b2d4-45fc-9194-9de212c9410f\") " pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.529745 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.529722    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:20.561288 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.561268    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t74nq"
Jul 22 02:22:20.599244 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:22:20.599219    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd103b8_b2d4_45fc_9194_9de212c9410f.slice/crio-2368f78d333fc06531126e49f67c100763077fdc0776d2f1eee90ee021cb3e4d WatchSource:0}: Error finding container 2368f78d333fc06531126e49f67c100763077fdc0776d2f1eee90ee021cb3e4d: Status 404 returned error can't find the container with id 2368f78d333fc06531126e49f67c100763077fdc0776d2f1eee90ee021cb3e4d
Jul 22 02:22:20.629860 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:20.629825    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:21.126118 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:21.126087    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:21.238867 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:21.238826    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:21.255197 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:21.255160    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t74nq" event={"ID":"9dd103b8-b2d4-45fc-9194-9de212c9410f","Type":"ContainerStarted","Data":"2368f78d333fc06531126e49f67c100763077fdc0776d2f1eee90ee021cb3e4d"}
Jul 22 02:22:21.519642 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:21.519566    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:22.775514 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:22.775483    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:22.832771 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:22.832741    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:23.062429 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:23.062351    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:23.163993 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:23.163961    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:23.219184 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:23.219154    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:23.269853 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:23.269825    2565 generic.go:356] "Generic (PLEG): container finished" podID="9dd103b8-b2d4-45fc-9194-9de212c9410f" containerID="47d9ebfb912f17116ea246bd4058b9b43da156d16cf2f820c436360f0ab7e04c" exitCode=0
Jul 22 02:22:23.269985 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:23.269880    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t74nq" event={"ID":"9dd103b8-b2d4-45fc-9194-9de212c9410f","Type":"ContainerDied","Data":"47d9ebfb912f17116ea246bd4058b9b43da156d16cf2f820c436360f0ab7e04c"}
Jul 22 02:22:23.337964 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:23.337933    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:23.829165 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:23.829134    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:24.276648 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:24.276609    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t74nq" event={"ID":"9dd103b8-b2d4-45fc-9194-9de212c9410f","Type":"ContainerStarted","Data":"0701a516098044343c899f47145502d0bdf24c3854dd288e89e1941230fc3791"}
Jul 22 02:22:24.276814 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:24.276680    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t74nq" event={"ID":"9dd103b8-b2d4-45fc-9194-9de212c9410f","Type":"ContainerStarted","Data":"812858ccee5df8ac2d216f75f1e24cb4bb384f5f221b8470e7475d39151101f9"}
Jul 22 02:22:24.295227 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:24.295182    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t74nq" podStartSLOduration=2.634265018 podStartE2EDuration="4.295168129s" totalImagesPullingTime="1.556591106s" totalInitContainerRuntime="104.312005ms" isStatefulPod=true podCreationTimestamp="2026-07-22 02:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:22:24.295157016 +0000 UTC m=+95.143493593" watchObservedRunningTime="2026-07-22 02:22:24.295168129 +0000 UTC m=+95.143504694"
Jul 22 02:22:24.594017 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:24.593926    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:24.656363 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:24.656334    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:24.908597 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:24.908565    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:25.006750 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:25.006720    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:25.264283 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:25.264197    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:25.778449 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:25.778412    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:25.837640 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:25.837604    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:26.375166 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:26.375134    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:26.445045 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:26.445014    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:28.564464 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:28.564429    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:28.664566 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:28.664524    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:33.965435 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:33.965395    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:34.021977 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:34.021940    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:36.294823 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:36.294787    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:36.393767 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:36.393733    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:36.489693 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:36.489664    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:38.207317 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:38.207293    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-r4hdg"
Jul 22 02:22:42.399385 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:42.399352    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:22:42.458383 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:22:42.458354    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:23:39.329576 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:23:39.329542    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:23:39.530396 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:23:39.530360    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:23:46.445486 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:23:46.445404    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:23:48.202394 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:23:48.202352    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:23:48.277251 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:23:48.277217    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:23:54.627438 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:23:54.627404    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:23:54.697020 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:23:54.696990    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:23:58.623369 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:23:58.623337    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:23:58.722789 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:23:58.722756    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:23:58.974327 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:23:58.974229    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:23:59.056295 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:23:59.056264    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:24:06.646407 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:24:06.646372    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:24:06.760335 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:24:06.760303    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:24:09.761247 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:24:09.761212    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:24:09.850780 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:24:09.850751    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:24:09.961538 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:24:09.961508    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:24:18.992183 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:24:18.992143    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:24:19.088933 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:24:19.088894    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:24:19.935394 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:24:19.935357    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:24:20.019136 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:24:20.019105    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:24:51.030761 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:24:51.030721    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:24:51.214449 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:24:51.214416    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:14.533306 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:14.533224    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:15.532251 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:15.532214    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:18.536424 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:18.536391    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:18.636301 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:18.636271    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:23.804006 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:23.803969    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:25.066145 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:25.066114    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:25.973187 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:25.973151    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:26.523541 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:26.523514    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:32.738138 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:32.738100    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:32.815897 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:32.815864    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:35.324284 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:35.324253    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:37.277667 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:37.277635    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:37.377131 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:37.377105    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:43.810169 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:43.810123    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:43.904840 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:43.904815    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:45.522771 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:45.522742    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:48.338890 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:48.338841    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:49.580693 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:49.580664    2565 kubelet.go:1648] "Image garbage collection succeeded"
Jul 22 02:25:52.928478 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:52.928443    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:59.225249 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:59.225218    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:59.325032 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:59.325003    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:25:59.692080 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:25:59.692043    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:00.091577 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.091508    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:00.188260 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.188236    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:00.227950 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.227927    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"]
Jul 22 02:26:00.230239 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.230222    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"
Jul 22 02:26:00.233243 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.233226    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-z24rf\""
Jul 22 02:26:00.234296 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.234279    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 22 02:26:00.234361 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.234334    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 22 02:26:00.234398 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.234369    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 22 02:26:00.234756 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.234735    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 22 02:26:00.247624 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.247604    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"]
Jul 22 02:26:00.306506 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.306466    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/27e667ef-ad44-4c46-9a62-7c34985a6713-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fcd77b786-zzhbg\" (UID: \"27e667ef-ad44-4c46-9a62-7c34985a6713\") " pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"
Jul 22 02:26:00.306679 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.306520    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27e667ef-ad44-4c46-9a62-7c34985a6713-tmp\") pod \"opendatahub-operator-controller-manager-5fcd77b786-zzhbg\" (UID: \"27e667ef-ad44-4c46-9a62-7c34985a6713\") " pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"
Jul 22 02:26:00.306679 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.306581    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27e667ef-ad44-4c46-9a62-7c34985a6713-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fcd77b786-zzhbg\" (UID: \"27e667ef-ad44-4c46-9a62-7c34985a6713\") " pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"
Jul 22 02:26:00.306679 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.306612    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8rt9\" (UniqueName: \"kubernetes.io/projected/27e667ef-ad44-4c46-9a62-7c34985a6713-kube-api-access-v8rt9\") pod \"opendatahub-operator-controller-manager-5fcd77b786-zzhbg\" (UID: \"27e667ef-ad44-4c46-9a62-7c34985a6713\") " pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"
Jul 22 02:26:00.306679 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.306643    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/27e667ef-ad44-4c46-9a62-7c34985a6713-manifests\") pod \"opendatahub-operator-controller-manager-5fcd77b786-zzhbg\" (UID: \"27e667ef-ad44-4c46-9a62-7c34985a6713\") " pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"
Jul 22 02:26:00.407896 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.407842    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/27e667ef-ad44-4c46-9a62-7c34985a6713-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fcd77b786-zzhbg\" (UID: \"27e667ef-ad44-4c46-9a62-7c34985a6713\") " pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"
Jul 22 02:26:00.408081 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.407916    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27e667ef-ad44-4c46-9a62-7c34985a6713-tmp\") pod \"opendatahub-operator-controller-manager-5fcd77b786-zzhbg\" (UID: \"27e667ef-ad44-4c46-9a62-7c34985a6713\") " pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"
Jul 22 02:26:00.408081 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.407952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27e667ef-ad44-4c46-9a62-7c34985a6713-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fcd77b786-zzhbg\" (UID: \"27e667ef-ad44-4c46-9a62-7c34985a6713\") " pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"
Jul 22 02:26:00.408081 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.407969    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8rt9\" (UniqueName: \"kubernetes.io/projected/27e667ef-ad44-4c46-9a62-7c34985a6713-kube-api-access-v8rt9\") pod \"opendatahub-operator-controller-manager-5fcd77b786-zzhbg\" (UID: \"27e667ef-ad44-4c46-9a62-7c34985a6713\") " pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"
Jul 22 02:26:00.408081 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.408009    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/27e667ef-ad44-4c46-9a62-7c34985a6713-manifests\") pod \"opendatahub-operator-controller-manager-5fcd77b786-zzhbg\" (UID: \"27e667ef-ad44-4c46-9a62-7c34985a6713\") " pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"
Jul 22 02:26:00.408438 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.408408    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/27e667ef-ad44-4c46-9a62-7c34985a6713-manifests\") pod \"opendatahub-operator-controller-manager-5fcd77b786-zzhbg\" (UID: \"27e667ef-ad44-4c46-9a62-7c34985a6713\") " pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"
Jul 22 02:26:00.410192 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.410171    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27e667ef-ad44-4c46-9a62-7c34985a6713-tmp\") pod \"opendatahub-operator-controller-manager-5fcd77b786-zzhbg\" (UID: \"27e667ef-ad44-4c46-9a62-7c34985a6713\") " pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"
Jul 22 02:26:00.410413 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.410394    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/27e667ef-ad44-4c46-9a62-7c34985a6713-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fcd77b786-zzhbg\" (UID: \"27e667ef-ad44-4c46-9a62-7c34985a6713\") " pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"
Jul 22 02:26:00.410456 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.410396    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27e667ef-ad44-4c46-9a62-7c34985a6713-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fcd77b786-zzhbg\" (UID: \"27e667ef-ad44-4c46-9a62-7c34985a6713\") " pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"
Jul 22 02:26:00.417941 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.417920    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8rt9\" (UniqueName: \"kubernetes.io/projected/27e667ef-ad44-4c46-9a62-7c34985a6713-kube-api-access-v8rt9\") pod \"opendatahub-operator-controller-manager-5fcd77b786-zzhbg\" (UID: \"27e667ef-ad44-4c46-9a62-7c34985a6713\") " pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"
Jul 22 02:26:00.443616 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.443594    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:00.540475 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.540452    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"
Jul 22 02:26:00.574151 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.574125    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:00.696220 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.696149    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:00.696354 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.696328    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"]
Jul 22 02:26:00.703839 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:26:00.703815    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e667ef_ad44_4c46_9a62_7c34985a6713.slice/crio-b8d2e61d6ad53059e9b7098a812b6e77e5b700afa33408853df432bdfd64071e WatchSource:0}: Error finding container b8d2e61d6ad53059e9b7098a812b6e77e5b700afa33408853df432bdfd64071e: Status 404 returned error can't find the container with id b8d2e61d6ad53059e9b7098a812b6e77e5b700afa33408853df432bdfd64071e
Jul 22 02:26:00.705424 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.705407    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 02:26:00.843155 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.843124    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:00.943759 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.943727    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:00.976516 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:00.976433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg" event={"ID":"27e667ef-ad44-4c46-9a62-7c34985a6713","Type":"ContainerStarted","Data":"b8d2e61d6ad53059e9b7098a812b6e77e5b700afa33408853df432bdfd64071e"}
Jul 22 02:26:01.043375 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:01.043350    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:01.144210 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:01.144174    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:03.989446 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:03.989414    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg" event={"ID":"27e667ef-ad44-4c46-9a62-7c34985a6713","Type":"ContainerStarted","Data":"76fbb4eea53436fe6c3fc55a61e736e9dbdd6263ec0ceef62e2cec4ab9d7daae"}
Jul 22 02:26:04.994410 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:04.994378    2565 generic.go:356] "Generic (PLEG): container finished" podID="27e667ef-ad44-4c46-9a62-7c34985a6713" containerID="76fbb4eea53436fe6c3fc55a61e736e9dbdd6263ec0ceef62e2cec4ab9d7daae" exitCode=0
Jul 22 02:26:04.994829 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:04.994471    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg" event={"ID":"27e667ef-ad44-4c46-9a62-7c34985a6713","Type":"ContainerDied","Data":"76fbb4eea53436fe6c3fc55a61e736e9dbdd6263ec0ceef62e2cec4ab9d7daae"}
Jul 22 02:26:05.999299 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:05.999266    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg" event={"ID":"27e667ef-ad44-4c46-9a62-7c34985a6713","Type":"ContainerStarted","Data":"3c9f9f653cb04c28d3c430e3ca20fc68300fcc31c47ba1783010016b44dfd6ed"}
Jul 22 02:26:05.999719 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:05.999501    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"
Jul 22 02:26:06.025081 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:06.025033    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg" podStartSLOduration=1.9474613029999999 podStartE2EDuration="6.025018065s" totalImagesPullingTime="3.076295336s" totalInitContainerRuntime="1.001261426s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:26:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:26:06.023307084 +0000 UTC m=+316.871643659" watchObservedRunningTime="2026-07-22 02:26:06.025018065 +0000 UTC m=+316.873354638"
Jul 22 02:26:11.343080 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:11.343041    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:11.446098 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:11.446061    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:17.005032 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:17.005004    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5fcd77b786-zzhbg"
Jul 22 02:26:18.625361 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:18.625332    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:18.735259 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:18.735234    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:19.454474 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:19.454444    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:19.682204 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:19.682176    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:20.350435 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:20.350398    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qhj4q"]
Jul 22 02:26:20.355755 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:20.355732    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-qhj4q"
Jul 22 02:26:20.360912 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:20.360878    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72crr\" (UniqueName: \"kubernetes.io/projected/77956685-f072-4905-a803-ebef81ed3145-kube-api-access-72crr\") pod \"odh-model-controller-7998c67dc4-qhj4q\" (UID: \"77956685-f072-4905-a803-ebef81ed3145\") " pod="opendatahub/odh-model-controller-7998c67dc4-qhj4q"
Jul 22 02:26:20.361075 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:20.360942    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77956685-f072-4905-a803-ebef81ed3145-cert\") pod \"odh-model-controller-7998c67dc4-qhj4q\" (UID: \"77956685-f072-4905-a803-ebef81ed3145\") " pod="opendatahub/odh-model-controller-7998c67dc4-qhj4q"
Jul 22 02:26:20.361450 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:20.361427    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 22 02:26:20.361450 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:20.361445    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-bd9vk\""
Jul 22 02:26:20.380864 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:20.380827    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qhj4q"]
Jul 22 02:26:20.461925 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:20.461891    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72crr\" (UniqueName: \"kubernetes.io/projected/77956685-f072-4905-a803-ebef81ed3145-kube-api-access-72crr\") pod \"odh-model-controller-7998c67dc4-qhj4q\" (UID: \"77956685-f072-4905-a803-ebef81ed3145\") " pod="opendatahub/odh-model-controller-7998c67dc4-qhj4q"
Jul 22 02:26:20.462088 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:20.461938    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77956685-f072-4905-a803-ebef81ed3145-cert\") pod \"odh-model-controller-7998c67dc4-qhj4q\" (UID: \"77956685-f072-4905-a803-ebef81ed3145\") " pod="opendatahub/odh-model-controller-7998c67dc4-qhj4q"
Jul 22 02:26:20.462088 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:26:20.462059    2565 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 22 02:26:20.462227 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:26:20.462117    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77956685-f072-4905-a803-ebef81ed3145-cert podName:77956685-f072-4905-a803-ebef81ed3145 nodeName:}" failed. No retries permitted until 2026-07-22 02:26:20.962099955 +0000 UTC m=+331.810436508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/77956685-f072-4905-a803-ebef81ed3145-cert") pod "odh-model-controller-7998c67dc4-qhj4q" (UID: "77956685-f072-4905-a803-ebef81ed3145") : secret "odh-model-controller-webhook-cert" not found
Jul 22 02:26:20.482785 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:20.482743    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:20.487691 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:20.487665    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72crr\" (UniqueName: \"kubernetes.io/projected/77956685-f072-4905-a803-ebef81ed3145-kube-api-access-72crr\") pod \"odh-model-controller-7998c67dc4-qhj4q\" (UID: \"77956685-f072-4905-a803-ebef81ed3145\") " pod="opendatahub/odh-model-controller-7998c67dc4-qhj4q"
Jul 22 02:26:20.698556 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:20.698524    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:20.939388 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:20.939356    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:20.966606 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:20.966530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77956685-f072-4905-a803-ebef81ed3145-cert\") pod \"odh-model-controller-7998c67dc4-qhj4q\" (UID: \"77956685-f072-4905-a803-ebef81ed3145\") " pod="opendatahub/odh-model-controller-7998c67dc4-qhj4q"
Jul 22 02:26:20.968949 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:20.968914    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77956685-f072-4905-a803-ebef81ed3145-cert\") pod \"odh-model-controller-7998c67dc4-qhj4q\" (UID: \"77956685-f072-4905-a803-ebef81ed3145\") " pod="opendatahub/odh-model-controller-7998c67dc4-qhj4q"
Jul 22 02:26:21.213159 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:21.213122    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:21.266267 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:21.266173    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-qhj4q"
Jul 22 02:26:21.448865 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:21.448807    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qhj4q"]
Jul 22 02:26:21.454264 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:26:21.454231    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77956685_f072_4905_a803_ebef81ed3145.slice/crio-44585d911678a2c537d5ad7207b1eace04dd29e1d35c80150421c75eb90ea888 WatchSource:0}: Error finding container 44585d911678a2c537d5ad7207b1eace04dd29e1d35c80150421c75eb90ea888: Status 404 returned error can't find the container with id 44585d911678a2c537d5ad7207b1eace04dd29e1d35c80150421c75eb90ea888
Jul 22 02:26:21.929065 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:21.929033    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ckcnw"]
Jul 22 02:26:21.931918 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:21.931895    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ckcnw"
Jul 22 02:26:21.935729 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:21.935707    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 22 02:26:21.974146 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:21.974019    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvs7c\" (UniqueName: \"kubernetes.io/projected/a1fe2929-e2c8-4231-ba3d-547a7b982853-kube-api-access-pvs7c\") pod \"model-serving-api-54d89bd5fc-ckcnw\" (UID: \"a1fe2929-e2c8-4231-ba3d-547a7b982853\") " pod="opendatahub/model-serving-api-54d89bd5fc-ckcnw"
Jul 22 02:26:21.974146 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:21.974101    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1fe2929-e2c8-4231-ba3d-547a7b982853-tls-certs\") pod \"model-serving-api-54d89bd5fc-ckcnw\" (UID: \"a1fe2929-e2c8-4231-ba3d-547a7b982853\") " pod="opendatahub/model-serving-api-54d89bd5fc-ckcnw"
Jul 22 02:26:22.000620 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:22.000549    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ckcnw"]
Jul 22 02:26:22.012871 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:22.012722    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:22.054192 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:22.054149    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qhj4q" event={"ID":"77956685-f072-4905-a803-ebef81ed3145","Type":"ContainerStarted","Data":"44585d911678a2c537d5ad7207b1eace04dd29e1d35c80150421c75eb90ea888"}
Jul 22 02:26:22.075482 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:22.075450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1fe2929-e2c8-4231-ba3d-547a7b982853-tls-certs\") pod \"model-serving-api-54d89bd5fc-ckcnw\" (UID: \"a1fe2929-e2c8-4231-ba3d-547a7b982853\") " pod="opendatahub/model-serving-api-54d89bd5fc-ckcnw"
Jul 22 02:26:22.075635 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:22.075554    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvs7c\" (UniqueName: \"kubernetes.io/projected/a1fe2929-e2c8-4231-ba3d-547a7b982853-kube-api-access-pvs7c\") pod \"model-serving-api-54d89bd5fc-ckcnw\" (UID: \"a1fe2929-e2c8-4231-ba3d-547a7b982853\") " pod="opendatahub/model-serving-api-54d89bd5fc-ckcnw"
Jul 22 02:26:22.078283 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:22.078259    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1fe2929-e2c8-4231-ba3d-547a7b982853-tls-certs\") pod \"model-serving-api-54d89bd5fc-ckcnw\" (UID: \"a1fe2929-e2c8-4231-ba3d-547a7b982853\") " pod="opendatahub/model-serving-api-54d89bd5fc-ckcnw"
Jul 22 02:26:22.089165 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:22.089138    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvs7c\" (UniqueName: \"kubernetes.io/projected/a1fe2929-e2c8-4231-ba3d-547a7b982853-kube-api-access-pvs7c\") pod \"model-serving-api-54d89bd5fc-ckcnw\" (UID: \"a1fe2929-e2c8-4231-ba3d-547a7b982853\") " pod="opendatahub/model-serving-api-54d89bd5fc-ckcnw"
Jul 22 02:26:22.132903 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:22.132824    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:22.245901 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:22.245786    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ckcnw"
Jul 22 02:26:22.402251 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:22.402217    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ckcnw"]
Jul 22 02:26:22.406119 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:26:22.406087    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1fe2929_e2c8_4231_ba3d_547a7b982853.slice/crio-47e2a3a0fa1fb3a0623a42ab13a29d734e72aa5e4208a03d6700c616bee7baec WatchSource:0}: Error finding container 47e2a3a0fa1fb3a0623a42ab13a29d734e72aa5e4208a03d6700c616bee7baec: Status 404 returned error can't find the container with id 47e2a3a0fa1fb3a0623a42ab13a29d734e72aa5e4208a03d6700c616bee7baec
Jul 22 02:26:23.059798 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:23.059756    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ckcnw" event={"ID":"a1fe2929-e2c8-4231-ba3d-547a7b982853","Type":"ContainerStarted","Data":"47e2a3a0fa1fb3a0623a42ab13a29d734e72aa5e4208a03d6700c616bee7baec"}
Jul 22 02:26:25.070189 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:25.070142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qhj4q" event={"ID":"77956685-f072-4905-a803-ebef81ed3145","Type":"ContainerStarted","Data":"ae5bf0d087185a191e095b921ef22f0a5ce43bb641bf0a0478f27974b541477e"}
Jul 22 02:26:25.070655 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:25.070209    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-qhj4q"
Jul 22 02:26:25.071567 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:25.071543    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ckcnw" event={"ID":"a1fe2929-e2c8-4231-ba3d-547a7b982853","Type":"ContainerStarted","Data":"d819e510ba015aa8c9b3aad1fbe9f2f86faf9b3dbe383b66f2edb0789e272b6c"}
Jul 22 02:26:25.071656 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:25.071595    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-ckcnw"
Jul 22 02:26:25.091009 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:25.090962    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-qhj4q" podStartSLOduration=2.007510826 podStartE2EDuration="5.090946927s" totalImagesPullingTime="3.083436101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:26:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:26:25.089918701 +0000 UTC m=+335.938255275" watchObservedRunningTime="2026-07-22 02:26:25.090946927 +0000 UTC m=+335.939283500"
Jul 22 02:26:25.109060 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:25.109011    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-ckcnw" podStartSLOduration=1.9790155 podStartE2EDuration="4.108983031s" totalImagesPullingTime="2.129967531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:26:25.108501429 +0000 UTC m=+335.956838002" watchObservedRunningTime="2026-07-22 02:26:25.108983031 +0000 UTC m=+335.957319605"
Jul 22 02:26:25.934890 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:25.934840    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:25.967525 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:25.967493    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-79676bf5cc-fgwxm"]
Jul 22 02:26:25.970637 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:25.970620    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79676bf5cc-fgwxm"
Jul 22 02:26:25.975064 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:25.975023    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 22 02:26:25.975232 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:25.975142    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 22 02:26:25.975298 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:25.975269    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 22 02:26:25.975626 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:25.975602    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 22 02:26:25.987480 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:25.987453    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79676bf5cc-fgwxm"]
Jul 22 02:26:26.009839 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:26.009812    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d7f7d87-3443-42f1-b8c6-c07d6f272922-tmp\") pod \"kube-auth-proxy-79676bf5cc-fgwxm\" (UID: \"5d7f7d87-3443-42f1-b8c6-c07d6f272922\") " pod="openshift-ingress/kube-auth-proxy-79676bf5cc-fgwxm"
Jul 22 02:26:26.009965 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:26.009933    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d7f7d87-3443-42f1-b8c6-c07d6f272922-tls-certs\") pod \"kube-auth-proxy-79676bf5cc-fgwxm\" (UID: \"5d7f7d87-3443-42f1-b8c6-c07d6f272922\") " pod="openshift-ingress/kube-auth-proxy-79676bf5cc-fgwxm"
Jul 22 02:26:26.010022 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:26.009974    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6sgj\" (UniqueName: \"kubernetes.io/projected/5d7f7d87-3443-42f1-b8c6-c07d6f272922-kube-api-access-k6sgj\") pod \"kube-auth-proxy-79676bf5cc-fgwxm\" (UID: \"5d7f7d87-3443-42f1-b8c6-c07d6f272922\") " pod="openshift-ingress/kube-auth-proxy-79676bf5cc-fgwxm"
Jul 22 02:26:26.111192 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:26.111158    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d7f7d87-3443-42f1-b8c6-c07d6f272922-tmp\") pod \"kube-auth-proxy-79676bf5cc-fgwxm\" (UID: \"5d7f7d87-3443-42f1-b8c6-c07d6f272922\") " pod="openshift-ingress/kube-auth-proxy-79676bf5cc-fgwxm"
Jul 22 02:26:26.111655 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:26.111202    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d7f7d87-3443-42f1-b8c6-c07d6f272922-tls-certs\") pod \"kube-auth-proxy-79676bf5cc-fgwxm\" (UID: \"5d7f7d87-3443-42f1-b8c6-c07d6f272922\") " pod="openshift-ingress/kube-auth-proxy-79676bf5cc-fgwxm"
Jul 22 02:26:26.111655 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:26.111221    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6sgj\" (UniqueName: \"kubernetes.io/projected/5d7f7d87-3443-42f1-b8c6-c07d6f272922-kube-api-access-k6sgj\") pod \"kube-auth-proxy-79676bf5cc-fgwxm\" (UID: \"5d7f7d87-3443-42f1-b8c6-c07d6f272922\") " pod="openshift-ingress/kube-auth-proxy-79676bf5cc-fgwxm"
Jul 22 02:26:26.111655 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:26:26.111319    2565 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 22 02:26:26.111655 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:26:26.111391    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7f7d87-3443-42f1-b8c6-c07d6f272922-tls-certs podName:5d7f7d87-3443-42f1-b8c6-c07d6f272922 nodeName:}" failed. No retries permitted until 2026-07-22 02:26:26.611371206 +0000 UTC m=+337.459707768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5d7f7d87-3443-42f1-b8c6-c07d6f272922-tls-certs") pod "kube-auth-proxy-79676bf5cc-fgwxm" (UID: "5d7f7d87-3443-42f1-b8c6-c07d6f272922") : secret "kube-auth-proxy-tls" not found
Jul 22 02:26:26.113504 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:26.113474    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d7f7d87-3443-42f1-b8c6-c07d6f272922-tmp\") pod \"kube-auth-proxy-79676bf5cc-fgwxm\" (UID: \"5d7f7d87-3443-42f1-b8c6-c07d6f272922\") " pod="openshift-ingress/kube-auth-proxy-79676bf5cc-fgwxm"
Jul 22 02:26:26.121527 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:26.121495    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6sgj\" (UniqueName: \"kubernetes.io/projected/5d7f7d87-3443-42f1-b8c6-c07d6f272922-kube-api-access-k6sgj\") pod \"kube-auth-proxy-79676bf5cc-fgwxm\" (UID: \"5d7f7d87-3443-42f1-b8c6-c07d6f272922\") " pod="openshift-ingress/kube-auth-proxy-79676bf5cc-fgwxm"
Jul 22 02:26:26.472137 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:26.472107    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:26.616171 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:26.616134    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d7f7d87-3443-42f1-b8c6-c07d6f272922-tls-certs\") pod \"kube-auth-proxy-79676bf5cc-fgwxm\" (UID: \"5d7f7d87-3443-42f1-b8c6-c07d6f272922\") " pod="openshift-ingress/kube-auth-proxy-79676bf5cc-fgwxm"
Jul 22 02:26:26.618473 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:26.618449    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d7f7d87-3443-42f1-b8c6-c07d6f272922-tls-certs\") pod \"kube-auth-proxy-79676bf5cc-fgwxm\" (UID: \"5d7f7d87-3443-42f1-b8c6-c07d6f272922\") " pod="openshift-ingress/kube-auth-proxy-79676bf5cc-fgwxm"
Jul 22 02:26:26.888475 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:26.888439    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79676bf5cc-fgwxm"
Jul 22 02:26:27.048167 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.048135    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79676bf5cc-fgwxm"]
Jul 22 02:26:27.056446 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:26:27.056408    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7f7d87_3443_42f1_b8c6_c07d6f272922.slice/crio-61ebb02268f3f292a8cdabfed9dabc755784326a5b45bc0d706db2fc47dda03a WatchSource:0}: Error finding container 61ebb02268f3f292a8cdabfed9dabc755784326a5b45bc0d706db2fc47dda03a: Status 404 returned error can't find the container with id 61ebb02268f3f292a8cdabfed9dabc755784326a5b45bc0d706db2fc47dda03a
Jul 22 02:26:27.079121 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.079091    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79676bf5cc-fgwxm" event={"ID":"5d7f7d87-3443-42f1-b8c6-c07d6f272922","Type":"ContainerStarted","Data":"61ebb02268f3f292a8cdabfed9dabc755784326a5b45bc0d706db2fc47dda03a"}
Jul 22 02:26:27.687576 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.687539    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:27.725876 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.725826    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-d664x"]
Jul 22 02:26:27.729265 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.729245    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.732158 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.732135    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 22 02:26:27.732290 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.732202    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-hxk99\""
Jul 22 02:26:27.732290 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.732213    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 22 02:26:27.732412 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.732139    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 22 02:26:27.732616 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.732595    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 22 02:26:27.743094 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.743070    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-d664x"]
Jul 22 02:26:27.803203 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.803179    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:27.826449 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.826420    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86214211-09d7-4a51-ba89-668d79c33c29-local-certs\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.826602 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.826457    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86214211-09d7-4a51-ba89-668d79c33c29-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.826602 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.826483    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86214211-09d7-4a51-ba89-668d79c33c29-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.826602 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.826535    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86214211-09d7-4a51-ba89-668d79c33c29-istio-token\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.826602 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.826582    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86214211-09d7-4a51-ba89-668d79c33c29-cacerts\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.826793 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.826606    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86214211-09d7-4a51-ba89-668d79c33c29-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.826793 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.826660    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxdvh\" (UniqueName: \"kubernetes.io/projected/86214211-09d7-4a51-ba89-668d79c33c29-kube-api-access-jxdvh\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.927908 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.927868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86214211-09d7-4a51-ba89-668d79c33c29-local-certs\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.928084 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.927917    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86214211-09d7-4a51-ba89-668d79c33c29-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.928084 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.927949    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86214211-09d7-4a51-ba89-668d79c33c29-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.928084 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.927979    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86214211-09d7-4a51-ba89-668d79c33c29-istio-token\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.928084 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.928058    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86214211-09d7-4a51-ba89-668d79c33c29-cacerts\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.928286 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.928105    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86214211-09d7-4a51-ba89-668d79c33c29-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.928286 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.928175    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxdvh\" (UniqueName: \"kubernetes.io/projected/86214211-09d7-4a51-ba89-668d79c33c29-kube-api-access-jxdvh\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.929045 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.929018    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86214211-09d7-4a51-ba89-668d79c33c29-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.930955 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.930928    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86214211-09d7-4a51-ba89-668d79c33c29-cacerts\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.931058 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.930977    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86214211-09d7-4a51-ba89-668d79c33c29-local-certs\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.931058 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.930975    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86214211-09d7-4a51-ba89-668d79c33c29-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.932152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.932133    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86214211-09d7-4a51-ba89-668d79c33c29-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.937897 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.937834    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86214211-09d7-4a51-ba89-668d79c33c29-istio-token\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:27.938443 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:27.938401    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxdvh\" (UniqueName: \"kubernetes.io/projected/86214211-09d7-4a51-ba89-668d79c33c29-kube-api-access-jxdvh\") pod \"istiod-openshift-gateway-7574459656-d664x\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:28.041823 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:28.041791    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:28.219423 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:28.219323    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-d664x"]
Jul 22 02:26:28.223480 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:26:28.223450    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86214211_09d7_4a51_ba89_668d79c33c29.slice/crio-a7a8e128f72da87486e3e774a4510707533255ab3867ee622f039a05960a689c WatchSource:0}: Error finding container a7a8e128f72da87486e3e774a4510707533255ab3867ee622f039a05960a689c: Status 404 returned error can't find the container with id a7a8e128f72da87486e3e774a4510707533255ab3867ee622f039a05960a689c
Jul 22 02:26:29.090390 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:29.090349    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x" event={"ID":"86214211-09d7-4a51-ba89-668d79c33c29","Type":"ContainerStarted","Data":"a7a8e128f72da87486e3e774a4510707533255ab3867ee622f039a05960a689c"}
Jul 22 02:26:29.319818 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:29.319780    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:30.096109 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:30.096071    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79676bf5cc-fgwxm" event={"ID":"5d7f7d87-3443-42f1-b8c6-c07d6f272922","Type":"ContainerStarted","Data":"d7230e09a67b485ea9eba2ec4b89e1c619e01308058030eba51a63fe1339251d"}
Jul 22 02:26:30.142612 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:30.142557    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-79676bf5cc-fgwxm" podStartSLOduration=3.047402618 podStartE2EDuration="5.142542882s" totalImagesPullingTime="2.095140264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:26:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:26:30.129191462 +0000 UTC m=+340.977528035" watchObservedRunningTime="2026-07-22 02:26:30.142542882 +0000 UTC m=+340.990879455"
Jul 22 02:26:32.219484 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.219452    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:32.317925 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.317889    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-flrgk"]
Jul 22 02:26:32.321336 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.321313    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-flrgk"
Jul 22 02:26:32.321452 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.321326    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:26:32.321452 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.321431    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:26:32.329903 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.329842    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-t4k4b\""
Jul 22 02:26:32.330011 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.329981    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 22 02:26:32.341100 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.340844    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-flrgk"]
Jul 22 02:26:32.345567 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.345541    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fpw79"]
Jul 22 02:26:32.350215 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.350197    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpw79"
Jul 22 02:26:32.355875 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.355834    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 22 02:26:32.355954 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.355873    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-rv8r2\""
Jul 22 02:26:32.366533 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.366512    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec-cert\") pod \"kserve-controller-manager-d6f4b65b8-flrgk\" (UID: \"9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-flrgk"
Jul 22 02:26:32.366634 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.366559    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbrqz\" (UniqueName: \"kubernetes.io/projected/9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec-kube-api-access-lbrqz\") pod \"kserve-controller-manager-d6f4b65b8-flrgk\" (UID: \"9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-flrgk"
Jul 22 02:26:32.373058 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.373035    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fpw79"]
Jul 22 02:26:32.384657 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.384637    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:32.467351 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.467318    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec-cert\") pod \"kserve-controller-manager-d6f4b65b8-flrgk\" (UID: \"9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-flrgk"
Jul 22 02:26:32.467351 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.467355    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9541f8fd-025b-4524-873f-7fc567fcc58a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fpw79\" (UID: \"9541f8fd-025b-4524-873f-7fc567fcc58a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpw79"
Jul 22 02:26:32.467578 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.467377    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhsnp\" (UniqueName: \"kubernetes.io/projected/9541f8fd-025b-4524-873f-7fc567fcc58a-kube-api-access-nhsnp\") pod \"llmisvc-controller-manager-777c6bd7d5-fpw79\" (UID: \"9541f8fd-025b-4524-873f-7fc567fcc58a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpw79"
Jul 22 02:26:32.467578 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.467398    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbrqz\" (UniqueName: \"kubernetes.io/projected/9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec-kube-api-access-lbrqz\") pod \"kserve-controller-manager-d6f4b65b8-flrgk\" (UID: \"9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-flrgk"
Jul 22 02:26:32.467578 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:26:32.467477    2565 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 22 02:26:32.467578 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:26:32.467542    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec-cert podName:9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec nodeName:}" failed. No retries permitted until 2026-07-22 02:26:32.967525297 +0000 UTC m=+343.815861850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec-cert") pod "kserve-controller-manager-d6f4b65b8-flrgk" (UID: "9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec") : secret "kserve-webhook-server-cert" not found
Jul 22 02:26:32.485320 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.485254    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbrqz\" (UniqueName: \"kubernetes.io/projected/9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec-kube-api-access-lbrqz\") pod \"kserve-controller-manager-d6f4b65b8-flrgk\" (UID: \"9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-flrgk"
Jul 22 02:26:32.509898 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.509863    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:32.568700 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.568662    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9541f8fd-025b-4524-873f-7fc567fcc58a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fpw79\" (UID: \"9541f8fd-025b-4524-873f-7fc567fcc58a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpw79"
Jul 22 02:26:32.568700 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.568711    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhsnp\" (UniqueName: \"kubernetes.io/projected/9541f8fd-025b-4524-873f-7fc567fcc58a-kube-api-access-nhsnp\") pod \"llmisvc-controller-manager-777c6bd7d5-fpw79\" (UID: \"9541f8fd-025b-4524-873f-7fc567fcc58a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpw79"
Jul 22 02:26:32.571559 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.571531    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9541f8fd-025b-4524-873f-7fc567fcc58a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fpw79\" (UID: \"9541f8fd-025b-4524-873f-7fc567fcc58a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpw79"
Jul 22 02:26:32.620929 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.620895    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhsnp\" (UniqueName: \"kubernetes.io/projected/9541f8fd-025b-4524-873f-7fc567fcc58a-kube-api-access-nhsnp\") pod \"llmisvc-controller-manager-777c6bd7d5-fpw79\" (UID: \"9541f8fd-025b-4524-873f-7fc567fcc58a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpw79"
Jul 22 02:26:32.645334 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.645308    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:32.666317 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.666298    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpw79"
Jul 22 02:26:32.815354 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.815313    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fpw79"]
Jul 22 02:26:32.819062 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:26:32.819040    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9541f8fd_025b_4524_873f_7fc567fcc58a.slice/crio-e1d2a6ee306916d6cc9cdc0575478c9561ea387c532f680f7ab49198e7cbe1db WatchSource:0}: Error finding container e1d2a6ee306916d6cc9cdc0575478c9561ea387c532f680f7ab49198e7cbe1db: Status 404 returned error can't find the container with id e1d2a6ee306916d6cc9cdc0575478c9561ea387c532f680f7ab49198e7cbe1db
Jul 22 02:26:32.972967 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.972933    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec-cert\") pod \"kserve-controller-manager-d6f4b65b8-flrgk\" (UID: \"9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-flrgk"
Jul 22 02:26:32.975268 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:32.975246    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec-cert\") pod \"kserve-controller-manager-d6f4b65b8-flrgk\" (UID: \"9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-flrgk"
Jul 22 02:26:33.111279 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:33.111248    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x" event={"ID":"86214211-09d7-4a51-ba89-668d79c33c29","Type":"ContainerStarted","Data":"8b62d07429fe0d073da0c528bf7cfa26dbded6df31cf422dc9a57ebbd90db020"}
Jul 22 02:26:33.111458 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:33.111436    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:33.112589 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:33.112565    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpw79" event={"ID":"9541f8fd-025b-4524-873f-7fc567fcc58a","Type":"ContainerStarted","Data":"e1d2a6ee306916d6cc9cdc0575478c9561ea387c532f680f7ab49198e7cbe1db"}
Jul 22 02:26:33.113139 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:33.113111    2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-d664x container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 22 02:26:33.113239 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:33.113162    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x" podUID="86214211-09d7-4a51-ba89-668d79c33c29" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 22 02:26:33.143291 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:33.143246    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x" podStartSLOduration=2.048269049 podStartE2EDuration="6.143233332s" totalImagesPullingTime="4.094964283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:26:33.141255257 +0000 UTC m=+343.989591831" watchObservedRunningTime="2026-07-22 02:26:33.143233332 +0000 UTC m=+343.991569907"
Jul 22 02:26:33.232183 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:33.232156    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-flrgk"
Jul 22 02:26:33.333602 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:33.333490    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:33.415204 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:33.415115    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-flrgk"]
Jul 22 02:26:33.432840 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:26:33.432810    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b22e1aa_35d2_43f2_b9a0_60fa2546d1ec.slice/crio-c074c3300130cbbf2b4e96b5844df57b6dd24ca45997a165ec0a1325cf72d297 WatchSource:0}: Error finding container c074c3300130cbbf2b4e96b5844df57b6dd24ca45997a165ec0a1325cf72d297: Status 404 returned error can't find the container with id c074c3300130cbbf2b4e96b5844df57b6dd24ca45997a165ec0a1325cf72d297
Jul 22 02:26:33.465122 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:33.465086    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:33.807562 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:33.807486    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:33.913938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:33.913905    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:34.119916 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:34.119876    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-flrgk" event={"ID":"9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec","Type":"ContainerStarted","Data":"c074c3300130cbbf2b4e96b5844df57b6dd24ca45997a165ec0a1325cf72d297"}
Jul 22 02:26:34.120735 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:34.120705    2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-d664x container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 22 02:26:34.120894 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:34.120770    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x" podUID="86214211-09d7-4a51-ba89-668d79c33c29" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 22 02:26:34.931507 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:34.928353    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"]
Jul 22 02:26:34.936967 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:34.936935    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:34.943242 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:34.943216    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-pdc4t\""
Jul 22 02:26:34.959665 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:34.959633    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"]
Jul 22 02:26:34.974029 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:34.973407    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:34.993616 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:34.993583    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaae1329-63c3-460e-afcd-114656c1f9e1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:34.993779 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:34.993681    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaae1329-63c3-460e-afcd-114656c1f9e1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:34.993779 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:34.993744    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9kbx\" (UniqueName: \"kubernetes.io/projected/aaae1329-63c3-460e-afcd-114656c1f9e1-kube-api-access-k9kbx\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:34.993920 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:34.993784    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaae1329-63c3-460e-afcd-114656c1f9e1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:34.993920 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:34.993807    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaae1329-63c3-460e-afcd-114656c1f9e1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:34.993920 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:34.993891    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaae1329-63c3-460e-afcd-114656c1f9e1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:34.994078 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:34.993942    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaae1329-63c3-460e-afcd-114656c1f9e1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:34.994078 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:34.993975    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaae1329-63c3-460e-afcd-114656c1f9e1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:34.994078 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:34.994010    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaae1329-63c3-460e-afcd-114656c1f9e1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:35.095365 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.095328    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaae1329-63c3-460e-afcd-114656c1f9e1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:35.095623 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.095387    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9kbx\" (UniqueName: \"kubernetes.io/projected/aaae1329-63c3-460e-afcd-114656c1f9e1-kube-api-access-k9kbx\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:35.095623 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.095426    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaae1329-63c3-460e-afcd-114656c1f9e1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:35.095623 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.095453    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaae1329-63c3-460e-afcd-114656c1f9e1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:35.095623 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.095484    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaae1329-63c3-460e-afcd-114656c1f9e1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:35.095623 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.095515    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaae1329-63c3-460e-afcd-114656c1f9e1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:35.095623 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.095543    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaae1329-63c3-460e-afcd-114656c1f9e1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:35.095623 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.095576    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaae1329-63c3-460e-afcd-114656c1f9e1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:35.095623 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.095611    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaae1329-63c3-460e-afcd-114656c1f9e1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:35.096028 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.095785    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaae1329-63c3-460e-afcd-114656c1f9e1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:35.096118 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.096093    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaae1329-63c3-460e-afcd-114656c1f9e1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:35.096188 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.096164    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaae1329-63c3-460e-afcd-114656c1f9e1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:35.096604 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.096581    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaae1329-63c3-460e-afcd-114656c1f9e1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:35.096696 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.096611    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaae1329-63c3-460e-afcd-114656c1f9e1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:35.098533 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.098491    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaae1329-63c3-460e-afcd-114656c1f9e1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:35.098913 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.098829    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaae1329-63c3-460e-afcd-114656c1f9e1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:35.116577 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.116545    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaae1329-63c3-460e-afcd-114656c1f9e1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:35.124573 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.124542    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9kbx\" (UniqueName: \"kubernetes.io/projected/aaae1329-63c3-460e-afcd-114656c1f9e1-kube-api-access-k9kbx\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ck5mw\" (UID: \"aaae1329-63c3-460e-afcd-114656c1f9e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:35.251022 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.250937    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:35.365730 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.365616    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:35.682116 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.682083    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:26:35.962208 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:35.962184    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"]
Jul 22 02:26:35.964027 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:26:35.964002    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaae1329_63c3_460e_afcd_114656c1f9e1.slice/crio-d436b5357365c7e80ac25b724a2a6b3684db4bd569726f9af4286bc9957c9f46 WatchSource:0}: Error finding container d436b5357365c7e80ac25b724a2a6b3684db4bd569726f9af4286bc9957c9f46: Status 404 returned error can't find the container with id d436b5357365c7e80ac25b724a2a6b3684db4bd569726f9af4286bc9957c9f46
Jul 22 02:26:36.077390 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:36.077365    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-qhj4q"
Jul 22 02:26:36.079177 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:36.079157    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-ckcnw"
Jul 22 02:26:36.142473 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:36.142423    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-flrgk" event={"ID":"9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec","Type":"ContainerStarted","Data":"ee680a3cbab6c3122f54b39bd432813ef7c7950a5f14eac4025d69340e6fbe4e"}
Jul 22 02:26:36.143536 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:36.143321    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-flrgk"
Jul 22 02:26:36.144393 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:36.144360    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpw79" event={"ID":"9541f8fd-025b-4524-873f-7fc567fcc58a","Type":"ContainerStarted","Data":"d9f57a7a2e02034e8ff61dbfde7ee33ec9ea3bf7ec2f2deaa0b3a2801c43109f"}
Jul 22 02:26:36.144512 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:36.144421    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpw79"
Jul 22 02:26:36.145598 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:36.145576    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw" event={"ID":"aaae1329-63c3-460e-afcd-114656c1f9e1","Type":"ContainerStarted","Data":"d436b5357365c7e80ac25b724a2a6b3684db4bd569726f9af4286bc9957c9f46"}
Jul 22 02:26:36.191101 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:36.191042    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpw79" podStartSLOduration=1.195661504 podStartE2EDuration="4.191021747s" totalImagesPullingTime="2.995360243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:26:36.18936317 +0000 UTC m=+347.037699745" watchObservedRunningTime="2026-07-22 02:26:36.191021747 +0000 UTC m=+347.039358321"
Jul 22 02:26:36.191731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:36.191691    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-flrgk" podStartSLOduration=1.80481425 podStartE2EDuration="4.191680018s" totalImagesPullingTime="2.386865768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:26:36.167476084 +0000 UTC m=+347.015812688" watchObservedRunningTime="2026-07-22 02:26:36.191680018 +0000 UTC m=+347.040016592"
Jul 22 02:26:37.121326 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.121295    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:37.270176 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.270131    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-d664x"]
Jul 22 02:26:37.270355 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.270324    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x" podUID="86214211-09d7-4a51-ba89-668d79c33c29" containerName="discovery" containerID="cri-o://8b62d07429fe0d073da0c528bf7cfa26dbded6df31cf422dc9a57ebbd90db020" gracePeriod=30
Jul 22 02:26:37.542563 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.542526    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:37.619203 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.619166    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86214211-09d7-4a51-ba89-668d79c33c29-local-certs\") pod \"86214211-09d7-4a51-ba89-668d79c33c29\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") "
Jul 22 02:26:37.619384 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.619215    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86214211-09d7-4a51-ba89-668d79c33c29-cacerts\") pod \"86214211-09d7-4a51-ba89-668d79c33c29\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") "
Jul 22 02:26:37.619384 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.619257    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxdvh\" (UniqueName: \"kubernetes.io/projected/86214211-09d7-4a51-ba89-668d79c33c29-kube-api-access-jxdvh\") pod \"86214211-09d7-4a51-ba89-668d79c33c29\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") "
Jul 22 02:26:37.619384 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.619296    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86214211-09d7-4a51-ba89-668d79c33c29-istio-csr-ca-configmap\") pod \"86214211-09d7-4a51-ba89-668d79c33c29\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") "
Jul 22 02:26:37.619384 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.619356    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86214211-09d7-4a51-ba89-668d79c33c29-istio-token\") pod \"86214211-09d7-4a51-ba89-668d79c33c29\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") "
Jul 22 02:26:37.619601 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.619395    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86214211-09d7-4a51-ba89-668d79c33c29-istio-kubeconfig\") pod \"86214211-09d7-4a51-ba89-668d79c33c29\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") "
Jul 22 02:26:37.619601 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.619422    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86214211-09d7-4a51-ba89-668d79c33c29-istio-csr-dns-cert\") pod \"86214211-09d7-4a51-ba89-668d79c33c29\" (UID: \"86214211-09d7-4a51-ba89-668d79c33c29\") "
Jul 22 02:26:37.619892 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.619810    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86214211-09d7-4a51-ba89-668d79c33c29-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "86214211-09d7-4a51-ba89-668d79c33c29" (UID: "86214211-09d7-4a51-ba89-668d79c33c29"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:26:37.622070 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.622047    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86214211-09d7-4a51-ba89-668d79c33c29-cacerts" (OuterVolumeSpecName: "cacerts") pod "86214211-09d7-4a51-ba89-668d79c33c29" (UID: "86214211-09d7-4a51-ba89-668d79c33c29"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:26:37.622070 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.622059    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86214211-09d7-4a51-ba89-668d79c33c29-local-certs" (OuterVolumeSpecName: "local-certs") pod "86214211-09d7-4a51-ba89-668d79c33c29" (UID: "86214211-09d7-4a51-ba89-668d79c33c29"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:26:37.622200 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.622124    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86214211-09d7-4a51-ba89-668d79c33c29-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "86214211-09d7-4a51-ba89-668d79c33c29" (UID: "86214211-09d7-4a51-ba89-668d79c33c29"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:26:37.622501 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.622454    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86214211-09d7-4a51-ba89-668d79c33c29-kube-api-access-jxdvh" (OuterVolumeSpecName: "kube-api-access-jxdvh") pod "86214211-09d7-4a51-ba89-668d79c33c29" (UID: "86214211-09d7-4a51-ba89-668d79c33c29"). InnerVolumeSpecName "kube-api-access-jxdvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:26:37.622799 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.622775    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86214211-09d7-4a51-ba89-668d79c33c29-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "86214211-09d7-4a51-ba89-668d79c33c29" (UID: "86214211-09d7-4a51-ba89-668d79c33c29"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:26:37.623264 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.623241    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86214211-09d7-4a51-ba89-668d79c33c29-istio-token" (OuterVolumeSpecName: "istio-token") pod "86214211-09d7-4a51-ba89-668d79c33c29" (UID: "86214211-09d7-4a51-ba89-668d79c33c29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:26:37.720533 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.720443    2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86214211-09d7-4a51-ba89-668d79c33c29-istio-csr-ca-configmap\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:26:37.720533 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.720477    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86214211-09d7-4a51-ba89-668d79c33c29-istio-token\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:26:37.720533 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.720491    2565 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86214211-09d7-4a51-ba89-668d79c33c29-istio-kubeconfig\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:26:37.720533 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.720504    2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86214211-09d7-4a51-ba89-668d79c33c29-istio-csr-dns-cert\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:26:37.720533 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.720517    2565 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86214211-09d7-4a51-ba89-668d79c33c29-local-certs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:26:37.720533 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.720531    2565 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86214211-09d7-4a51-ba89-668d79c33c29-cacerts\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:26:37.720533 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:37.720544    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxdvh\" (UniqueName: \"kubernetes.io/projected/86214211-09d7-4a51-ba89-668d79c33c29-kube-api-access-jxdvh\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:26:38.160593 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:38.160558    2565 generic.go:356] "Generic (PLEG): container finished" podID="86214211-09d7-4a51-ba89-668d79c33c29" containerID="8b62d07429fe0d073da0c528bf7cfa26dbded6df31cf422dc9a57ebbd90db020" exitCode=0
Jul 22 02:26:38.161076 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:38.160612    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x"
Jul 22 02:26:38.161076 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:38.160658    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x" event={"ID":"86214211-09d7-4a51-ba89-668d79c33c29","Type":"ContainerDied","Data":"8b62d07429fe0d073da0c528bf7cfa26dbded6df31cf422dc9a57ebbd90db020"}
Jul 22 02:26:38.161076 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:38.160706    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d664x" event={"ID":"86214211-09d7-4a51-ba89-668d79c33c29","Type":"ContainerDied","Data":"a7a8e128f72da87486e3e774a4510707533255ab3867ee622f039a05960a689c"}
Jul 22 02:26:38.161076 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:38.160733    2565 scope.go:117] "RemoveContainer" containerID="8b62d07429fe0d073da0c528bf7cfa26dbded6df31cf422dc9a57ebbd90db020"
Jul 22 02:26:38.181376 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:38.181350    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-d664x"]
Jul 22 02:26:38.183870 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:38.183834    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-d664x"]
Jul 22 02:26:38.803203 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:38.803028    2565 scope.go:117] "RemoveContainer" containerID="8b62d07429fe0d073da0c528bf7cfa26dbded6df31cf422dc9a57ebbd90db020"
Jul 22 02:26:38.803380 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:26:38.803359    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b62d07429fe0d073da0c528bf7cfa26dbded6df31cf422dc9a57ebbd90db020\": container with ID starting with 8b62d07429fe0d073da0c528bf7cfa26dbded6df31cf422dc9a57ebbd90db020 not found: ID does not exist" containerID="8b62d07429fe0d073da0c528bf7cfa26dbded6df31cf422dc9a57ebbd90db020"
Jul 22 02:26:38.803423 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:38.803390    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b62d07429fe0d073da0c528bf7cfa26dbded6df31cf422dc9a57ebbd90db020"} err="failed to get container status \"8b62d07429fe0d073da0c528bf7cfa26dbded6df31cf422dc9a57ebbd90db020\": rpc error: code = NotFound desc = could not find container \"8b62d07429fe0d073da0c528bf7cfa26dbded6df31cf422dc9a57ebbd90db020\": container with ID starting with 8b62d07429fe0d073da0c528bf7cfa26dbded6df31cf422dc9a57ebbd90db020 not found: ID does not exist"
Jul 22 02:26:38.861138 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:38.861103    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:26:38.861233 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:38.861171    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:26:38.861233 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:38.861197    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:26:39.172197 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:39.172164    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw" event={"ID":"aaae1329-63c3-460e-afcd-114656c1f9e1","Type":"ContainerStarted","Data":"66473c4fd256bbfddff106584d0ba6da65eca5821a35a97b2f056225d6f76cd1"}
Jul 22 02:26:39.198869 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:39.198794    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw" podStartSLOduration=2.303970604 podStartE2EDuration="5.198781394s" totalImagesPullingTime="2.89481079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:26:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:26:39.197334263 +0000 UTC m=+350.045670836" watchObservedRunningTime="2026-07-22 02:26:39.198781394 +0000 UTC m=+350.047117969"
Jul 22 02:26:39.251210 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:39.251182    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:39.255604 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:39.255584    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:39.726632 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:39.726597    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86214211-09d7-4a51-ba89-668d79c33c29" path="/var/lib/kubelet/pods/86214211-09d7-4a51-ba89-668d79c33c29/volumes"
Jul 22 02:26:40.176313 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:40.176283    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:40.177253 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:40.177231    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ck5mw"
Jul 22 02:26:49.687126 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:26:49.687086    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:27:07.155188 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:07.155158    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fpw79"
Jul 22 02:27:07.158242 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:07.158226    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-flrgk"
Jul 22 02:27:20.930114 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:20.930075    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:27:21.033114 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.033076    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:27:21.065732 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.065704    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fgpml"]
Jul 22 02:27:21.066005 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.065993    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86214211-09d7-4a51-ba89-668d79c33c29" containerName="discovery"
Jul 22 02:27:21.066056 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.066007    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="86214211-09d7-4a51-ba89-668d79c33c29" containerName="discovery"
Jul 22 02:27:21.066094 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.066068    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="86214211-09d7-4a51-ba89-668d79c33c29" containerName="discovery"
Jul 22 02:27:21.070527 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.070499    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fgpml"
Jul 22 02:27:21.074184 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.074163    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-px492\""
Jul 22 02:27:21.074866 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.074832    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 22 02:27:21.074971 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.074899    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 22 02:27:21.089734 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.089709    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fgpml"]
Jul 22 02:27:21.146776 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.146747    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:27:21.147785 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.147762    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv6dh\" (UniqueName: \"kubernetes.io/projected/f74168ed-a5b8-44dd-a04f-29073f55de04-kube-api-access-cv6dh\") pod \"kuadrant-operator-catalog-fgpml\" (UID: \"f74168ed-a5b8-44dd-a04f-29073f55de04\") " pod="kuadrant-system/kuadrant-operator-catalog-fgpml"
Jul 22 02:27:21.248372 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.248297    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv6dh\" (UniqueName: \"kubernetes.io/projected/f74168ed-a5b8-44dd-a04f-29073f55de04-kube-api-access-cv6dh\") pod \"kuadrant-operator-catalog-fgpml\" (UID: \"f74168ed-a5b8-44dd-a04f-29073f55de04\") " pod="kuadrant-system/kuadrant-operator-catalog-fgpml"
Jul 22 02:27:21.256625 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.256597    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv6dh\" (UniqueName: \"kubernetes.io/projected/f74168ed-a5b8-44dd-a04f-29073f55de04-kube-api-access-cv6dh\") pod \"kuadrant-operator-catalog-fgpml\" (UID: \"f74168ed-a5b8-44dd-a04f-29073f55de04\") " pod="kuadrant-system/kuadrant-operator-catalog-fgpml"
Jul 22 02:27:21.373362 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.373335    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fgpml"]
Jul 22 02:27:21.373559 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.373548    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fgpml"
Jul 22 02:27:21.502496 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.502429    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fgpml"]
Jul 22 02:27:21.505799 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:27:21.505774    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74168ed_a5b8_44dd_a04f_29073f55de04.slice/crio-1f09d5cfc66195cb502657a2d536489bb65e2db95b01f549a439be6146771729 WatchSource:0}: Error finding container 1f09d5cfc66195cb502657a2d536489bb65e2db95b01f549a439be6146771729: Status 404 returned error can't find the container with id 1f09d5cfc66195cb502657a2d536489bb65e2db95b01f549a439be6146771729
Jul 22 02:27:21.585377 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.585348    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hrkpx"]
Jul 22 02:27:21.587758 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.587743    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hrkpx"
Jul 22 02:27:21.600267 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.600247    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hrkpx"]
Jul 22 02:27:21.650401 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.650370    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k288\" (UniqueName: \"kubernetes.io/projected/b131391a-810c-4f76-be3e-4dfe4decba2c-kube-api-access-9k288\") pod \"kuadrant-operator-catalog-hrkpx\" (UID: \"b131391a-810c-4f76-be3e-4dfe4decba2c\") " pod="kuadrant-system/kuadrant-operator-catalog-hrkpx"
Jul 22 02:27:21.751144 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.751117    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k288\" (UniqueName: \"kubernetes.io/projected/b131391a-810c-4f76-be3e-4dfe4decba2c-kube-api-access-9k288\") pod \"kuadrant-operator-catalog-hrkpx\" (UID: \"b131391a-810c-4f76-be3e-4dfe4decba2c\") " pod="kuadrant-system/kuadrant-operator-catalog-hrkpx"
Jul 22 02:27:21.761761 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.761696    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k288\" (UniqueName: \"kubernetes.io/projected/b131391a-810c-4f76-be3e-4dfe4decba2c-kube-api-access-9k288\") pod \"kuadrant-operator-catalog-hrkpx\" (UID: \"b131391a-810c-4f76-be3e-4dfe4decba2c\") " pod="kuadrant-system/kuadrant-operator-catalog-hrkpx"
Jul 22 02:27:21.820386 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.820356    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:27:21.896676 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:21.896642    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hrkpx"
Jul 22 02:27:22.035365 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:22.035318    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:27:22.119815 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:22.119784    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hrkpx"]
Jul 22 02:27:22.192484 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:27:22.192439    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb131391a_810c_4f76_be3e_4dfe4decba2c.slice/crio-394cc51772f3eefa5703ea335783856c848aff3dd48a321a334ab2215a54a3ef WatchSource:0}: Error finding container 394cc51772f3eefa5703ea335783856c848aff3dd48a321a334ab2215a54a3ef: Status 404 returned error can't find the container with id 394cc51772f3eefa5703ea335783856c848aff3dd48a321a334ab2215a54a3ef
Jul 22 02:27:22.339428 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:22.339336    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hrkpx" event={"ID":"b131391a-810c-4f76-be3e-4dfe4decba2c","Type":"ContainerStarted","Data":"394cc51772f3eefa5703ea335783856c848aff3dd48a321a334ab2215a54a3ef"}
Jul 22 02:27:22.340654 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:22.340620    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fgpml" event={"ID":"f74168ed-a5b8-44dd-a04f-29073f55de04","Type":"ContainerStarted","Data":"1f09d5cfc66195cb502657a2d536489bb65e2db95b01f549a439be6146771729"}
Jul 22 02:27:23.346902 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:23.346863    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fgpml" event={"ID":"f74168ed-a5b8-44dd-a04f-29073f55de04","Type":"ContainerStarted","Data":"820f11791e843fb2c4b17ddd41793df6a7287cfd2a7240d7cba13259f0d58330"}
Jul 22 02:27:23.347333 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:23.346914    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-fgpml" podUID="f74168ed-a5b8-44dd-a04f-29073f55de04" containerName="registry-server" containerID="cri-o://820f11791e843fb2c4b17ddd41793df6a7287cfd2a7240d7cba13259f0d58330" gracePeriod=2
Jul 22 02:27:23.348325 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:23.348292    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hrkpx" event={"ID":"b131391a-810c-4f76-be3e-4dfe4decba2c","Type":"ContainerStarted","Data":"a014a333ab9f85b1a18608aaffcdd17324522492b9e97c995d9e05505ebcb391"}
Jul 22 02:27:23.362878 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:23.362801    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-fgpml" podStartSLOduration=0.697173007 podStartE2EDuration="2.362788242s" totalImagesPullingTime="1.665615235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:27:23.361816145 +0000 UTC m=+394.210152716" watchObservedRunningTime="2026-07-22 02:27:23.362788242 +0000 UTC m=+394.211124816"
Jul 22 02:27:23.378931 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:23.378885    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-hrkpx" podStartSLOduration=1.400298879 podStartE2EDuration="2.378872444s" totalImagesPullingTime="978.573565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:27:23.377270703 +0000 UTC m=+394.225607269" watchObservedRunningTime="2026-07-22 02:27:23.378872444 +0000 UTC m=+394.227209008"
Jul 22 02:27:23.592156 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:23.592134    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fgpml"
Jul 22 02:27:23.664837 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:23.664809    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv6dh\" (UniqueName: \"kubernetes.io/projected/f74168ed-a5b8-44dd-a04f-29073f55de04-kube-api-access-cv6dh\") pod \"f74168ed-a5b8-44dd-a04f-29073f55de04\" (UID: \"f74168ed-a5b8-44dd-a04f-29073f55de04\") "
Jul 22 02:27:23.666793 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:23.666763    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74168ed-a5b8-44dd-a04f-29073f55de04-kube-api-access-cv6dh" (OuterVolumeSpecName: "kube-api-access-cv6dh") pod "f74168ed-a5b8-44dd-a04f-29073f55de04" (UID: "f74168ed-a5b8-44dd-a04f-29073f55de04"). InnerVolumeSpecName "kube-api-access-cv6dh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:27:23.765917 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:23.765894    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv6dh\" (UniqueName: \"kubernetes.io/projected/f74168ed-a5b8-44dd-a04f-29073f55de04-kube-api-access-cv6dh\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:27:24.353768 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:24.353731    2565 generic.go:356] "Generic (PLEG): container finished" podID="f74168ed-a5b8-44dd-a04f-29073f55de04" containerID="820f11791e843fb2c4b17ddd41793df6a7287cfd2a7240d7cba13259f0d58330" exitCode=0
Jul 22 02:27:24.354236 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:24.353783    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fgpml"
Jul 22 02:27:24.354236 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:24.353816    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fgpml" event={"ID":"f74168ed-a5b8-44dd-a04f-29073f55de04","Type":"ContainerDied","Data":"820f11791e843fb2c4b17ddd41793df6a7287cfd2a7240d7cba13259f0d58330"}
Jul 22 02:27:24.354236 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:24.353879    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fgpml" event={"ID":"f74168ed-a5b8-44dd-a04f-29073f55de04","Type":"ContainerDied","Data":"1f09d5cfc66195cb502657a2d536489bb65e2db95b01f549a439be6146771729"}
Jul 22 02:27:24.354236 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:24.353901    2565 scope.go:117] "RemoveContainer" containerID="820f11791e843fb2c4b17ddd41793df6a7287cfd2a7240d7cba13259f0d58330"
Jul 22 02:27:24.362742 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:24.362726    2565 scope.go:117] "RemoveContainer" containerID="820f11791e843fb2c4b17ddd41793df6a7287cfd2a7240d7cba13259f0d58330"
Jul 22 02:27:24.363042 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:27:24.363024    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"820f11791e843fb2c4b17ddd41793df6a7287cfd2a7240d7cba13259f0d58330\": container with ID starting with 820f11791e843fb2c4b17ddd41793df6a7287cfd2a7240d7cba13259f0d58330 not found: ID does not exist" containerID="820f11791e843fb2c4b17ddd41793df6a7287cfd2a7240d7cba13259f0d58330"
Jul 22 02:27:24.363101 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:24.363050    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"820f11791e843fb2c4b17ddd41793df6a7287cfd2a7240d7cba13259f0d58330"} err="failed to get container status \"820f11791e843fb2c4b17ddd41793df6a7287cfd2a7240d7cba13259f0d58330\": rpc error: code = NotFound desc = could not find container \"820f11791e843fb2c4b17ddd41793df6a7287cfd2a7240d7cba13259f0d58330\": container with ID starting with 820f11791e843fb2c4b17ddd41793df6a7287cfd2a7240d7cba13259f0d58330 not found: ID does not exist"
Jul 22 02:27:24.369581 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:24.369560    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fgpml"]
Jul 22 02:27:24.373621 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:24.373603    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fgpml"]
Jul 22 02:27:25.725502 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:25.725466    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f74168ed-a5b8-44dd-a04f-29073f55de04" path="/var/lib/kubelet/pods/f74168ed-a5b8-44dd-a04f-29073f55de04/volumes"
Jul 22 02:27:31.897270 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:31.897234    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-hrkpx"
Jul 22 02:27:31.897700 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:31.897297    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-hrkpx"
Jul 22 02:27:31.919963 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:31.919939    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-hrkpx"
Jul 22 02:27:32.409395 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:32.409369    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-hrkpx"
Jul 22 02:27:44.830666 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:44.830634    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:27:44.932678 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:44.932649    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:27:45.081319 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:45.081241    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:27:47.758330 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:47.758300    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"]
Jul 22 02:27:47.758681 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:47.758570    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f74168ed-a5b8-44dd-a04f-29073f55de04" containerName="registry-server"
Jul 22 02:27:47.758681 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:47.758581    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f74168ed-a5b8-44dd-a04f-29073f55de04" containerName="registry-server"
Jul 22 02:27:47.758681 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:47.758639    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f74168ed-a5b8-44dd-a04f-29073f55de04" containerName="registry-server"
Jul 22 02:27:47.760463 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:47.760446    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"
Jul 22 02:27:47.764416 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:47.764396    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-7zldb\""
Jul 22 02:27:47.775740 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:47.775719    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"]
Jul 22 02:27:47.817358 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:47.817324    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nczhc\" (UniqueName: \"kubernetes.io/projected/14d45ee8-dda4-41d9-970e-220e48b5aad4-kube-api-access-nczhc\") pod \"limitador-operator-controller-manager-859d7b78bd-z4hn4\" (UID: \"14d45ee8-dda4-41d9-970e-220e48b5aad4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"
Jul 22 02:27:47.918013 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:47.917980    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nczhc\" (UniqueName: \"kubernetes.io/projected/14d45ee8-dda4-41d9-970e-220e48b5aad4-kube-api-access-nczhc\") pod \"limitador-operator-controller-manager-859d7b78bd-z4hn4\" (UID: \"14d45ee8-dda4-41d9-970e-220e48b5aad4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"
Jul 22 02:27:47.933527 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:47.933499    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nczhc\" (UniqueName: \"kubernetes.io/projected/14d45ee8-dda4-41d9-970e-220e48b5aad4-kube-api-access-nczhc\") pod \"limitador-operator-controller-manager-859d7b78bd-z4hn4\" (UID: \"14d45ee8-dda4-41d9-970e-220e48b5aad4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"
Jul 22 02:27:48.070958 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:48.070863    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"
Jul 22 02:27:48.195116 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:48.195092    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"]
Jul 22 02:27:48.197754 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:27:48.197728    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d45ee8_dda4_41d9_970e_220e48b5aad4.slice/crio-ed87f5863f84c132a73bb985d037aa0bca6e2cee5112761e343c6c9e5e458352 WatchSource:0}: Error finding container ed87f5863f84c132a73bb985d037aa0bca6e2cee5112761e343c6c9e5e458352: Status 404 returned error can't find the container with id ed87f5863f84c132a73bb985d037aa0bca6e2cee5112761e343c6c9e5e458352
Jul 22 02:27:48.453656 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:48.453622    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4" event={"ID":"14d45ee8-dda4-41d9-970e-220e48b5aad4","Type":"ContainerStarted","Data":"ed87f5863f84c132a73bb985d037aa0bca6e2cee5112761e343c6c9e5e458352"}
Jul 22 02:27:48.486832 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:48.486802    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:27:49.298539 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:49.298505    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:27:50.464374 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:50.464338    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4" event={"ID":"14d45ee8-dda4-41d9-970e-220e48b5aad4","Type":"ContainerStarted","Data":"57920acc0e32f8f63a9c9066064bcf22127a926d83268f46fe2001b5cf4d22bb"}
Jul 22 02:27:50.464764 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:50.464459    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"
Jul 22 02:27:50.485721 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:50.485675    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4" podStartSLOduration=1.845390873 podStartE2EDuration="3.485661646s" totalImagesPullingTime="1.640270773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:27:50.484958035 +0000 UTC m=+421.333294602" watchObservedRunningTime="2026-07-22 02:27:50.485661646 +0000 UTC m=+421.333998267"
Jul 22 02:27:52.032721 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:52.032685    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-w7vvk"]
Jul 22 02:27:52.036154 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:52.036137    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w7vvk"
Jul 22 02:27:52.038778 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:52.038754    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-t867w\""
Jul 22 02:27:52.049648 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:52.049618    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swz8g\" (UniqueName: \"kubernetes.io/projected/23999d7a-c2c0-421a-be84-13d16b314726-kube-api-access-swz8g\") pod \"authorino-operator-75cc8d5cb7-w7vvk\" (UID: \"23999d7a-c2c0-421a-be84-13d16b314726\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-w7vvk"
Jul 22 02:27:52.051034 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:52.051015    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-w7vvk"]
Jul 22 02:27:52.150241 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:52.150211    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swz8g\" (UniqueName: \"kubernetes.io/projected/23999d7a-c2c0-421a-be84-13d16b314726-kube-api-access-swz8g\") pod \"authorino-operator-75cc8d5cb7-w7vvk\" (UID: \"23999d7a-c2c0-421a-be84-13d16b314726\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-w7vvk"
Jul 22 02:27:52.162784 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:52.162754    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swz8g\" (UniqueName: \"kubernetes.io/projected/23999d7a-c2c0-421a-be84-13d16b314726-kube-api-access-swz8g\") pod \"authorino-operator-75cc8d5cb7-w7vvk\" (UID: \"23999d7a-c2c0-421a-be84-13d16b314726\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-w7vvk"
Jul 22 02:27:52.346283 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:52.346200    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w7vvk"
Jul 22 02:27:52.489392 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:52.489357    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-w7vvk"]
Jul 22 02:27:52.497602 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:27:52.497570    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23999d7a_c2c0_421a_be84_13d16b314726.slice/crio-a1f5de0fd59c862ead1b6178f78e290d2ff8b3b3b574a518f789890bcd182b22 WatchSource:0}: Error finding container a1f5de0fd59c862ead1b6178f78e290d2ff8b3b3b574a518f789890bcd182b22: Status 404 returned error can't find the container with id a1f5de0fd59c862ead1b6178f78e290d2ff8b3b3b574a518f789890bcd182b22
Jul 22 02:27:52.566181 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:52.566141    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:27:52.656507 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:52.656480    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:27:52.758314 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:52.758277    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:27:53.478788 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:53.478752    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w7vvk" event={"ID":"23999d7a-c2c0-421a-be84-13d16b314726","Type":"ContainerStarted","Data":"a1f5de0fd59c862ead1b6178f78e290d2ff8b3b3b574a518f789890bcd182b22"}
Jul 22 02:27:54.925225 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:54.925194    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:27:55.488193 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:55.488160    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w7vvk" event={"ID":"23999d7a-c2c0-421a-be84-13d16b314726","Type":"ContainerStarted","Data":"9ee385b9a4f7848ad7d40b90cd31cad849b9b4678d91847ad0e3d0aed6933664"}
Jul 22 02:27:55.488366 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:55.488227    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w7vvk"
Jul 22 02:27:55.511095 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:55.511051    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w7vvk" podStartSLOduration=1.499968026 podStartE2EDuration="3.511035927s" totalImagesPullingTime="2.011067901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:27:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:27:55.509249722 +0000 UTC m=+426.357586297" watchObservedRunningTime="2026-07-22 02:27:55.511035927 +0000 UTC m=+426.359372569"
Jul 22 02:27:56.971986 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:56.971950    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:27:57.043411 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:57.043375    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"]
Jul 22 02:27:57.046693 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:57.046677    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:27:57.049912 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:57.049891    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kqb8r\""
Jul 22 02:27:57.064690 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:57.064669    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"]
Jul 22 02:27:57.088125 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:57.088101    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9a9533d-6887-4fe4-b977-12b23ae8ccbf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rs2h8\" (UID: \"c9a9533d-6887-4fe4-b977-12b23ae8ccbf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:27:57.088212 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:57.088166    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5rn5\" (UniqueName: \"kubernetes.io/projected/c9a9533d-6887-4fe4-b977-12b23ae8ccbf-kube-api-access-k5rn5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rs2h8\" (UID: \"c9a9533d-6887-4fe4-b977-12b23ae8ccbf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:27:57.188817 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:57.188787    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9a9533d-6887-4fe4-b977-12b23ae8ccbf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rs2h8\" (UID: \"c9a9533d-6887-4fe4-b977-12b23ae8ccbf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:27:57.188991 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:57.188839    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5rn5\" (UniqueName: \"kubernetes.io/projected/c9a9533d-6887-4fe4-b977-12b23ae8ccbf-kube-api-access-k5rn5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rs2h8\" (UID: \"c9a9533d-6887-4fe4-b977-12b23ae8ccbf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:27:57.189185 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:57.189163    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9a9533d-6887-4fe4-b977-12b23ae8ccbf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rs2h8\" (UID: \"c9a9533d-6887-4fe4-b977-12b23ae8ccbf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:27:57.207594 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:57.207565    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5rn5\" (UniqueName: \"kubernetes.io/projected/c9a9533d-6887-4fe4-b977-12b23ae8ccbf-kube-api-access-k5rn5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rs2h8\" (UID: \"c9a9533d-6887-4fe4-b977-12b23ae8ccbf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:27:57.356336 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:57.356307    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:27:57.489906 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:57.489881    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"]
Jul 22 02:27:57.492330 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:27:57.492289    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a9533d_6887_4fe4_b977_12b23ae8ccbf.slice/crio-db06a45de4455ddb62f48d0b10e6c3a148d49ac3f6695189a637a1342575b402 WatchSource:0}: Error finding container db06a45de4455ddb62f48d0b10e6c3a148d49ac3f6695189a637a1342575b402: Status 404 returned error can't find the container with id db06a45de4455ddb62f48d0b10e6c3a148d49ac3f6695189a637a1342575b402
Jul 22 02:27:57.499195 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:57.499160    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8" event={"ID":"c9a9533d-6887-4fe4-b977-12b23ae8ccbf","Type":"ContainerStarted","Data":"db06a45de4455ddb62f48d0b10e6c3a148d49ac3f6695189a637a1342575b402"}
Jul 22 02:27:58.099166 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:27:58.099131    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:00.139760 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:00.139715    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:01.470529 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:01.470498    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"
Jul 22 02:28:01.518998 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:01.518963    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8" event={"ID":"c9a9533d-6887-4fe4-b977-12b23ae8ccbf","Type":"ContainerStarted","Data":"8a8b37ab461637b7c0cd7ab8832e43375f4a6aa8afecb0f7d93e9e89c45e9ed8"}
Jul 22 02:28:01.519187 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:01.519085    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:28:01.541471 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:01.541416    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8" podStartSLOduration=0.981175461 podStartE2EDuration="4.541402116s" totalImagesPullingTime="3.560226655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:01.540072092 +0000 UTC m=+432.388408667" watchObservedRunningTime="2026-07-22 02:28:01.541402116 +0000 UTC m=+432.389738689"
Jul 22 02:28:02.559738 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:02.559706    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:02.684309 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:02.684279    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:03.799618 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:03.799590    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:03.893284 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:03.893258    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:06.495619 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:06.495592    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w7vvk"
Jul 22 02:28:07.117123 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:07.117086    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:07.252159 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:07.252106    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:07.679696 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:07.679663    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:07.778103 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:07.778071    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:12.078820 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:12.078743    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:12.139235 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:12.139202    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:12.525829 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:12.525794    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:28:14.412446 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.412416    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"]
Jul 22 02:28:14.412872 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.412603    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8" podUID="c9a9533d-6887-4fe4-b977-12b23ae8ccbf" containerName="manager" containerID="cri-o://8a8b37ab461637b7c0cd7ab8832e43375f4a6aa8afecb0f7d93e9e89c45e9ed8" gracePeriod=2
Jul 22 02:28:14.421972 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.421949    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"]
Jul 22 02:28:14.428060 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:28:14.428025    2565 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"c9a9533d-6887-4fe4-b977-12b23ae8ccbf\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-rs2h8\": pods \"kuadrant-operator-controller-manager-8684c9ff49-rs2h8\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:28:14.443871 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.443820    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k"]
Jul 22 02:28:14.444331 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.444310    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9a9533d-6887-4fe4-b977-12b23ae8ccbf" containerName="manager"
Jul 22 02:28:14.444331 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.444338    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9a9533d-6887-4fe4-b977-12b23ae8ccbf" containerName="manager"
Jul 22 02:28:14.444486 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.444434    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9a9533d-6887-4fe4-b977-12b23ae8ccbf" containerName="manager"
Jul 22 02:28:14.447488 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.447470    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"]
Jul 22 02:28:14.447584 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.447561    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k"
Jul 22 02:28:14.447658 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.447635    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4" podUID="14d45ee8-dda4-41d9-970e-220e48b5aad4" containerName="manager" containerID="cri-o://57920acc0e32f8f63a9c9066064bcf22127a926d83268f46fe2001b5cf4d22bb" gracePeriod=2
Jul 22 02:28:14.468729 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.468706    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"]
Jul 22 02:28:14.470825 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.470798    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k"]
Jul 22 02:28:14.476628 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:28:14.476590    2565 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"14d45ee8-dda4-41d9-970e-220e48b5aad4\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-z4hn4\": pods \"limitador-operator-controller-manager-859d7b78bd-z4hn4\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"
Jul 22 02:28:14.479657 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:28:14.479611    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rs2h8\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="c9a9533d-6887-4fe4-b977-12b23ae8ccbf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:28:14.485509 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.483327    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grkxg"]
Jul 22 02:28:14.485509 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.483976    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14d45ee8-dda4-41d9-970e-220e48b5aad4" containerName="manager"
Jul 22 02:28:14.485509 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.483993    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="14d45ee8-dda4-41d9-970e-220e48b5aad4" containerName="manager"
Jul 22 02:28:14.485509 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.484146    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="14d45ee8-dda4-41d9-970e-220e48b5aad4" containerName="manager"
Jul 22 02:28:14.487921 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.487896    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grkxg"
Jul 22 02:28:14.499177 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.499158    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grkxg"]
Jul 22 02:28:14.512998 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:28:14.512967    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z4hn4\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="14d45ee8-dda4-41d9-970e-220e48b5aad4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"
Jul 22 02:28:14.515025 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:28:14.514988    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rs2h8\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="c9a9533d-6887-4fe4-b977-12b23ae8ccbf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:28:14.526339 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.526312    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-694hj\" (UniqueName: \"kubernetes.io/projected/1787f6b1-5328-45a5-9e0d-861344416277-kube-api-access-694hj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5fw7k\" (UID: \"1787f6b1-5328-45a5-9e0d-861344416277\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k"
Jul 22 02:28:14.526440 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.526379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1787f6b1-5328-45a5-9e0d-861344416277-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5fw7k\" (UID: \"1787f6b1-5328-45a5-9e0d-861344416277\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k"
Jul 22 02:28:14.553439 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:28:14.553404    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z4hn4\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="14d45ee8-dda4-41d9-970e-220e48b5aad4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"
Jul 22 02:28:14.555464 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:28:14.555438    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rs2h8\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="c9a9533d-6887-4fe4-b977-12b23ae8ccbf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:28:14.572822 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.572798    2565 generic.go:356] "Generic (PLEG): container finished" podID="14d45ee8-dda4-41d9-970e-220e48b5aad4" containerID="57920acc0e32f8f63a9c9066064bcf22127a926d83268f46fe2001b5cf4d22bb" exitCode=0
Jul 22 02:28:14.574881 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.574839    2565 generic.go:356] "Generic (PLEG): container finished" podID="c9a9533d-6887-4fe4-b977-12b23ae8ccbf" containerID="8a8b37ab461637b7c0cd7ab8832e43375f4a6aa8afecb0f7d93e9e89c45e9ed8" exitCode=0
Jul 22 02:28:14.627195 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.627169    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-694hj\" (UniqueName: \"kubernetes.io/projected/1787f6b1-5328-45a5-9e0d-861344416277-kube-api-access-694hj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5fw7k\" (UID: \"1787f6b1-5328-45a5-9e0d-861344416277\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k"
Jul 22 02:28:14.627324 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.627217    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1787f6b1-5328-45a5-9e0d-861344416277-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5fw7k\" (UID: \"1787f6b1-5328-45a5-9e0d-861344416277\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k"
Jul 22 02:28:14.627324 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.627253    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q65sk\" (UniqueName: \"kubernetes.io/projected/a627716e-c9d3-4b38-9494-1aebebcd1882-kube-api-access-q65sk\") pod \"limitador-operator-controller-manager-859d7b78bd-grkxg\" (UID: \"a627716e-c9d3-4b38-9494-1aebebcd1882\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grkxg"
Jul 22 02:28:14.627981 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.627949    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1787f6b1-5328-45a5-9e0d-861344416277-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5fw7k\" (UID: \"1787f6b1-5328-45a5-9e0d-861344416277\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k"
Jul 22 02:28:14.636593 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.636553    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-694hj\" (UniqueName: \"kubernetes.io/projected/1787f6b1-5328-45a5-9e0d-861344416277-kube-api-access-694hj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5fw7k\" (UID: \"1787f6b1-5328-45a5-9e0d-861344416277\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k"
Jul 22 02:28:14.677364 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.677344    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:28:14.679454 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:28:14.679425    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rs2h8\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="c9a9533d-6887-4fe4-b977-12b23ae8ccbf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:28:14.681355 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:28:14.681329    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z4hn4\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="14d45ee8-dda4-41d9-970e-220e48b5aad4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"
Jul 22 02:28:14.682909 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.682890    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"
Jul 22 02:28:14.684912 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:28:14.684890    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rs2h8\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="c9a9533d-6887-4fe4-b977-12b23ae8ccbf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:28:14.686814 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:28:14.686793    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z4hn4\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="14d45ee8-dda4-41d9-970e-220e48b5aad4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"
Jul 22 02:28:14.728345 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.728320    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9a9533d-6887-4fe4-b977-12b23ae8ccbf-extensions-socket-volume\") pod \"c9a9533d-6887-4fe4-b977-12b23ae8ccbf\" (UID: \"c9a9533d-6887-4fe4-b977-12b23ae8ccbf\") "
Jul 22 02:28:14.728438 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.728423    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nczhc\" (UniqueName: \"kubernetes.io/projected/14d45ee8-dda4-41d9-970e-220e48b5aad4-kube-api-access-nczhc\") pod \"14d45ee8-dda4-41d9-970e-220e48b5aad4\" (UID: \"14d45ee8-dda4-41d9-970e-220e48b5aad4\") "
Jul 22 02:28:14.728503 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.728458    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5rn5\" (UniqueName: \"kubernetes.io/projected/c9a9533d-6887-4fe4-b977-12b23ae8ccbf-kube-api-access-k5rn5\") pod \"c9a9533d-6887-4fe4-b977-12b23ae8ccbf\" (UID: \"c9a9533d-6887-4fe4-b977-12b23ae8ccbf\") "
Jul 22 02:28:14.728643 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.728629    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q65sk\" (UniqueName: \"kubernetes.io/projected/a627716e-c9d3-4b38-9494-1aebebcd1882-kube-api-access-q65sk\") pod \"limitador-operator-controller-manager-859d7b78bd-grkxg\" (UID: \"a627716e-c9d3-4b38-9494-1aebebcd1882\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grkxg"
Jul 22 02:28:14.728835 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.728811    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a9533d-6887-4fe4-b977-12b23ae8ccbf-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c9a9533d-6887-4fe4-b977-12b23ae8ccbf" (UID: "c9a9533d-6887-4fe4-b977-12b23ae8ccbf"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:28:14.730411 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.730380    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a9533d-6887-4fe4-b977-12b23ae8ccbf-kube-api-access-k5rn5" (OuterVolumeSpecName: "kube-api-access-k5rn5") pod "c9a9533d-6887-4fe4-b977-12b23ae8ccbf" (UID: "c9a9533d-6887-4fe4-b977-12b23ae8ccbf"). InnerVolumeSpecName "kube-api-access-k5rn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:28:14.730535 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.730444    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d45ee8-dda4-41d9-970e-220e48b5aad4-kube-api-access-nczhc" (OuterVolumeSpecName: "kube-api-access-nczhc") pod "14d45ee8-dda4-41d9-970e-220e48b5aad4" (UID: "14d45ee8-dda4-41d9-970e-220e48b5aad4"). InnerVolumeSpecName "kube-api-access-nczhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:28:14.745203 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.745173    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q65sk\" (UniqueName: \"kubernetes.io/projected/a627716e-c9d3-4b38-9494-1aebebcd1882-kube-api-access-q65sk\") pod \"limitador-operator-controller-manager-859d7b78bd-grkxg\" (UID: \"a627716e-c9d3-4b38-9494-1aebebcd1882\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grkxg"
Jul 22 02:28:14.819683 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.819657    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k"
Jul 22 02:28:14.825730 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.825705    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grkxg"
Jul 22 02:28:14.829632 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.829603    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nczhc\" (UniqueName: \"kubernetes.io/projected/14d45ee8-dda4-41d9-970e-220e48b5aad4-kube-api-access-nczhc\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:28:14.829698 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.829632    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5rn5\" (UniqueName: \"kubernetes.io/projected/c9a9533d-6887-4fe4-b977-12b23ae8ccbf-kube-api-access-k5rn5\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:28:14.829698 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.829648    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9a9533d-6887-4fe4-b977-12b23ae8ccbf-extensions-socket-volume\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:28:14.981383 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.981359    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k"]
Jul 22 02:28:14.982722 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:14.982693    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grkxg"]
Jul 22 02:28:14.982934 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:28:14.982915    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda627716e_c9d3_4b38_9494_1aebebcd1882.slice/crio-8d98ed082cc6f71e95fa35080a70e103cfc493982c43809d049deb3c16ee56cc WatchSource:0}: Error finding container 8d98ed082cc6f71e95fa35080a70e103cfc493982c43809d049deb3c16ee56cc: Status 404 returned error can't find the container with id 8d98ed082cc6f71e95fa35080a70e103cfc493982c43809d049deb3c16ee56cc
Jul 22 02:28:14.983448 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:28:14.983431    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1787f6b1_5328_45a5_9e0d_861344416277.slice/crio-7d4136d520f9fc223787df841bb37de6c7a8bba573068e07fa9a032212d0f3a9 WatchSource:0}: Error finding container 7d4136d520f9fc223787df841bb37de6c7a8bba573068e07fa9a032212d0f3a9: Status 404 returned error can't find the container with id 7d4136d520f9fc223787df841bb37de6c7a8bba573068e07fa9a032212d0f3a9
Jul 22 02:28:15.582227 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:15.582203    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:28:15.582648 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:15.582245    2565 scope.go:117] "RemoveContainer" containerID="8a8b37ab461637b7c0cd7ab8832e43375f4a6aa8afecb0f7d93e9e89c45e9ed8"
Jul 22 02:28:15.584055 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:15.584028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grkxg" event={"ID":"a627716e-c9d3-4b38-9494-1aebebcd1882","Type":"ContainerStarted","Data":"216d0fea2b7845455fbb3718d9f352ce0416182f87f9fb87f9427e1c251f97cb"}
Jul 22 02:28:15.584190 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:15.584081    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grkxg" event={"ID":"a627716e-c9d3-4b38-9494-1aebebcd1882","Type":"ContainerStarted","Data":"8d98ed082cc6f71e95fa35080a70e103cfc493982c43809d049deb3c16ee56cc"}
Jul 22 02:28:15.584190 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:15.584112    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grkxg"
Jul 22 02:28:15.584626 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:28:15.584596    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z4hn4\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="14d45ee8-dda4-41d9-970e-220e48b5aad4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"
Jul 22 02:28:15.585961 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:15.585874    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k" event={"ID":"1787f6b1-5328-45a5-9e0d-861344416277","Type":"ContainerStarted","Data":"abfca955df2953af48e95bd07ff3a496e85cc454ce3c8c96a7b4f6a0f4b3de17"}
Jul 22 02:28:15.585961 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:15.585925    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k"
Jul 22 02:28:15.585961 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:15.585950    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k" event={"ID":"1787f6b1-5328-45a5-9e0d-861344416277","Type":"ContainerStarted","Data":"7d4136d520f9fc223787df841bb37de6c7a8bba573068e07fa9a032212d0f3a9"}
Jul 22 02:28:15.586759 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:28:15.586602    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rs2h8\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="c9a9533d-6887-4fe4-b977-12b23ae8ccbf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:28:15.587831 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:15.587811    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"
Jul 22 02:28:15.588513 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:28:15.588488    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rs2h8\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="c9a9533d-6887-4fe4-b977-12b23ae8ccbf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:28:15.592974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:15.592960    2565 scope.go:117] "RemoveContainer" containerID="57920acc0e32f8f63a9c9066064bcf22127a926d83268f46fe2001b5cf4d22bb"
Jul 22 02:28:15.613379 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:15.613338    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k" podStartSLOduration=1.6133197460000002 podStartE2EDuration="1.613319746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:15.6118201 +0000 UTC m=+446.460156675" watchObservedRunningTime="2026-07-22 02:28:15.613319746 +0000 UTC m=+446.461656323"
Jul 22 02:28:15.635334 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:15.635296    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grkxg" podStartSLOduration=1.635283123 podStartE2EDuration="1.635283123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:15.633875317 +0000 UTC m=+446.482211892" watchObservedRunningTime="2026-07-22 02:28:15.635283123 +0000 UTC m=+446.483619696"
Jul 22 02:28:15.636065 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:28:15.636041    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z4hn4\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="14d45ee8-dda4-41d9-970e-220e48b5aad4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"
Jul 22 02:28:15.637961 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:28:15.637936    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z4hn4\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="14d45ee8-dda4-41d9-970e-220e48b5aad4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4hn4"
Jul 22 02:28:15.640065 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:28:15.640042    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-rs2h8\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="c9a9533d-6887-4fe4-b977-12b23ae8ccbf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rs2h8"
Jul 22 02:28:15.725761 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:15.725695    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14d45ee8-dda4-41d9-970e-220e48b5aad4" path="/var/lib/kubelet/pods/14d45ee8-dda4-41d9-970e-220e48b5aad4/volumes"
Jul 22 02:28:15.726061 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:15.726047    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9a9533d-6887-4fe4-b977-12b23ae8ccbf" path="/var/lib/kubelet/pods/c9a9533d-6887-4fe4-b977-12b23ae8ccbf/volumes"
Jul 22 02:28:20.275148 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:20.275108    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:20.374623 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:20.374592    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:26.076410 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:26.076375    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:26.176593 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:26.176550    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:26.596284 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:26.596257    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grkxg"
Jul 22 02:28:26.596443 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:26.596328    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k"
Jul 22 02:28:30.075137 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:30.075101    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:30.130039 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:30.130004    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:30.472887 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:30.472860    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:30.588314 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:30.588274    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:30.832020 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:30.831900    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k"]
Jul 22 02:28:30.832428 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:30.832149    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k" podUID="1787f6b1-5328-45a5-9e0d-861344416277" containerName="manager" containerID="cri-o://abfca955df2953af48e95bd07ff3a496e85cc454ce3c8c96a7b4f6a0f4b3de17" gracePeriod=10
Jul 22 02:28:31.069307 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:31.069282    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k"
Jul 22 02:28:31.154659 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:31.154626    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-694hj\" (UniqueName: \"kubernetes.io/projected/1787f6b1-5328-45a5-9e0d-861344416277-kube-api-access-694hj\") pod \"1787f6b1-5328-45a5-9e0d-861344416277\" (UID: \"1787f6b1-5328-45a5-9e0d-861344416277\") "
Jul 22 02:28:31.155152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:31.154669    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1787f6b1-5328-45a5-9e0d-861344416277-extensions-socket-volume\") pod \"1787f6b1-5328-45a5-9e0d-861344416277\" (UID: \"1787f6b1-5328-45a5-9e0d-861344416277\") "
Jul 22 02:28:31.155218 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:31.155144    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1787f6b1-5328-45a5-9e0d-861344416277-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1787f6b1-5328-45a5-9e0d-861344416277" (UID: "1787f6b1-5328-45a5-9e0d-861344416277"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:28:31.156652 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:31.156632    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1787f6b1-5328-45a5-9e0d-861344416277-kube-api-access-694hj" (OuterVolumeSpecName: "kube-api-access-694hj") pod "1787f6b1-5328-45a5-9e0d-861344416277" (UID: "1787f6b1-5328-45a5-9e0d-861344416277"). InnerVolumeSpecName "kube-api-access-694hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:28:31.255350 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:31.255315    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-694hj\" (UniqueName: \"kubernetes.io/projected/1787f6b1-5328-45a5-9e0d-861344416277-kube-api-access-694hj\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:28:31.255350 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:31.255344    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1787f6b1-5328-45a5-9e0d-861344416277-extensions-socket-volume\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:28:31.661014 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:31.660982    2565 generic.go:356] "Generic (PLEG): container finished" podID="1787f6b1-5328-45a5-9e0d-861344416277" containerID="abfca955df2953af48e95bd07ff3a496e85cc454ce3c8c96a7b4f6a0f4b3de17" exitCode=0
Jul 22 02:28:31.661184 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:31.661021    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k"
Jul 22 02:28:31.661184 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:31.661046    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k" event={"ID":"1787f6b1-5328-45a5-9e0d-861344416277","Type":"ContainerDied","Data":"abfca955df2953af48e95bd07ff3a496e85cc454ce3c8c96a7b4f6a0f4b3de17"}
Jul 22 02:28:31.661184 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:31.661094    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k" event={"ID":"1787f6b1-5328-45a5-9e0d-861344416277","Type":"ContainerDied","Data":"7d4136d520f9fc223787df841bb37de6c7a8bba573068e07fa9a032212d0f3a9"}
Jul 22 02:28:31.661184 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:31.661123    2565 scope.go:117] "RemoveContainer" containerID="abfca955df2953af48e95bd07ff3a496e85cc454ce3c8c96a7b4f6a0f4b3de17"
Jul 22 02:28:31.669701 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:31.669687    2565 scope.go:117] "RemoveContainer" containerID="abfca955df2953af48e95bd07ff3a496e85cc454ce3c8c96a7b4f6a0f4b3de17"
Jul 22 02:28:31.669951 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:28:31.669935    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abfca955df2953af48e95bd07ff3a496e85cc454ce3c8c96a7b4f6a0f4b3de17\": container with ID starting with abfca955df2953af48e95bd07ff3a496e85cc454ce3c8c96a7b4f6a0f4b3de17 not found: ID does not exist" containerID="abfca955df2953af48e95bd07ff3a496e85cc454ce3c8c96a7b4f6a0f4b3de17"
Jul 22 02:28:31.669997 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:31.669958    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abfca955df2953af48e95bd07ff3a496e85cc454ce3c8c96a7b4f6a0f4b3de17"} err="failed to get container status \"abfca955df2953af48e95bd07ff3a496e85cc454ce3c8c96a7b4f6a0f4b3de17\": rpc error: code = NotFound desc = could not find container \"abfca955df2953af48e95bd07ff3a496e85cc454ce3c8c96a7b4f6a0f4b3de17\": container with ID starting with abfca955df2953af48e95bd07ff3a496e85cc454ce3c8c96a7b4f6a0f4b3de17 not found: ID does not exist"
Jul 22 02:28:31.682721 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:31.682697    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k"]
Jul 22 02:28:31.689514 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:31.689496    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5fw7k"]
Jul 22 02:28:31.725446 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:31.725423    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1787f6b1-5328-45a5-9e0d-861344416277" path="/var/lib/kubelet/pods/1787f6b1-5328-45a5-9e0d-861344416277/volumes"
Jul 22 02:28:34.886554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:34.886495    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:34.959928 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:34.959899    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:43.878743 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:43.878706    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:43.933626 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:43.933600    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:47.204317 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.204280    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"]
Jul 22 02:28:47.204679 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.204579    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1787f6b1-5328-45a5-9e0d-861344416277" containerName="manager"
Jul 22 02:28:47.204679 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.204590    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1787f6b1-5328-45a5-9e0d-861344416277" containerName="manager"
Jul 22 02:28:47.204679 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.204643    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1787f6b1-5328-45a5-9e0d-861344416277" containerName="manager"
Jul 22 02:28:47.206604 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.206581    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.209416 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.209396    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cdgqc\""
Jul 22 02:28:47.224959 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.224935    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"]
Jul 22 02:28:47.242495 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.242468    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:47.381833 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.381794    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b40b006-204b-4291-901e-af512339c609-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.381833 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.381834    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b40b006-204b-4291-901e-af512339c609-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.382103 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.381910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b40b006-204b-4291-901e-af512339c609-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.382103 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.381958    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b40b006-204b-4291-901e-af512339c609-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.382103 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.382002    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b40b006-204b-4291-901e-af512339c609-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.382103 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.382024    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wn5r\" (UniqueName: \"kubernetes.io/projected/3b40b006-204b-4291-901e-af512339c609-kube-api-access-2wn5r\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.382103 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.382065    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b40b006-204b-4291-901e-af512339c609-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.382103 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.382094    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b40b006-204b-4291-901e-af512339c609-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.382320 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.382138    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b40b006-204b-4291-901e-af512339c609-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.427097 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.427055    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:47.483024 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.482935    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b40b006-204b-4291-901e-af512339c609-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.483024 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.482977    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b40b006-204b-4291-901e-af512339c609-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.483238 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.483105    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b40b006-204b-4291-901e-af512339c609-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.483238 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.483172    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b40b006-204b-4291-901e-af512339c609-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.483238 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.483222    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b40b006-204b-4291-901e-af512339c609-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.483391 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.483250    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wn5r\" (UniqueName: \"kubernetes.io/projected/3b40b006-204b-4291-901e-af512339c609-kube-api-access-2wn5r\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.483391 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.483293    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b40b006-204b-4291-901e-af512339c609-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.483391 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.483328    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b40b006-204b-4291-901e-af512339c609-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.483391 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.483346    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b40b006-204b-4291-901e-af512339c609-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.483586 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.483391    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b40b006-204b-4291-901e-af512339c609-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.483586 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.483451    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b40b006-204b-4291-901e-af512339c609-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.483586 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.483552    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b40b006-204b-4291-901e-af512339c609-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.483734 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.483680    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b40b006-204b-4291-901e-af512339c609-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.483972 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.483947    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b40b006-204b-4291-901e-af512339c609-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.485313 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.485289    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b40b006-204b-4291-901e-af512339c609-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.485786 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.485763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b40b006-204b-4291-901e-af512339c609-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.491486 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.491464    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b40b006-204b-4291-901e-af512339c609-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.491789 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.491769    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wn5r\" (UniqueName: \"kubernetes.io/projected/3b40b006-204b-4291-901e-af512339c609-kube-api-access-2wn5r\") pod \"maas-default-gateway-openshift-default-56654dcf7-xw2bp\" (UID: \"3b40b006-204b-4291-901e-af512339c609\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.516929 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.516899    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:47.649382 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.649357    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"]
Jul 22 02:28:47.651340 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:28:47.651301    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b40b006_204b_4291_901e_af512339c609.slice/crio-8ccde28b2656c155532b160c48d1da322c750c000bface3eea280ef2cec031bc WatchSource:0}: Error finding container 8ccde28b2656c155532b160c48d1da322c750c000bface3eea280ef2cec031bc: Status 404 returned error can't find the container with id 8ccde28b2656c155532b160c48d1da322c750c000bface3eea280ef2cec031bc
Jul 22 02:28:47.653436 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.653397    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:28:47.653525 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.653483    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:28:47.653567 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.653524    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:28:47.743540 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.743456    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp" event={"ID":"3b40b006-204b-4291-901e-af512339c609","Type":"ContainerStarted","Data":"9e889e43809e710e3f28609253817be26d3572ec7eb9ce40129ecd6f4f6693ef"}
Jul 22 02:28:47.743712 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.743547    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp" event={"ID":"3b40b006-204b-4291-901e-af512339c609","Type":"ContainerStarted","Data":"8ccde28b2656c155532b160c48d1da322c750c000bface3eea280ef2cec031bc"}
Jul 22 02:28:47.765273 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:47.765214    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp" podStartSLOduration=0.765198191 podStartE2EDuration="765.198191ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:47.763108856 +0000 UTC m=+478.611445430" watchObservedRunningTime="2026-07-22 02:28:47.765198191 +0000 UTC m=+478.613534765"
Jul 22 02:28:48.517859 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:48.517825    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:48.522433 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:48.522406    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:48.749187 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:48.749155    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:48.750022 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:48.750002    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:28:48.805158 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:48.805081    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-g5j4t"]
Jul 22 02:28:48.807546 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:48.807526    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-g5j4t"
Jul 22 02:28:48.809782 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:48.809757    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:48.812662 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:48.812640    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 22 02:28:48.812762 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:48.812688    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gbctx\""
Jul 22 02:28:48.819737 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:48.819710    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-g5j4t"]
Jul 22 02:28:48.896934 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:48.896899    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjkrw\" (UniqueName: \"kubernetes.io/projected/40c1b97c-3099-4d20-b35c-7b828db8d889-kube-api-access-xjkrw\") pod \"limitador-limitador-64c94fd64f-g5j4t\" (UID: \"40c1b97c-3099-4d20-b35c-7b828db8d889\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-g5j4t"
Jul 22 02:28:48.897115 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:48.897091    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40c1b97c-3099-4d20-b35c-7b828db8d889-config-file\") pod \"limitador-limitador-64c94fd64f-g5j4t\" (UID: \"40c1b97c-3099-4d20-b35c-7b828db8d889\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-g5j4t"
Jul 22 02:28:48.902988 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:48.902963    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-g5j4t"]
Jul 22 02:28:48.907292 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:48.907259    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:48.998570 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:48.998540    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjkrw\" (UniqueName: \"kubernetes.io/projected/40c1b97c-3099-4d20-b35c-7b828db8d889-kube-api-access-xjkrw\") pod \"limitador-limitador-64c94fd64f-g5j4t\" (UID: \"40c1b97c-3099-4d20-b35c-7b828db8d889\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-g5j4t"
Jul 22 02:28:48.998754 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:48.998586    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40c1b97c-3099-4d20-b35c-7b828db8d889-config-file\") pod \"limitador-limitador-64c94fd64f-g5j4t\" (UID: \"40c1b97c-3099-4d20-b35c-7b828db8d889\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-g5j4t"
Jul 22 02:28:48.999291 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:48.999266    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40c1b97c-3099-4d20-b35c-7b828db8d889-config-file\") pod \"limitador-limitador-64c94fd64f-g5j4t\" (UID: \"40c1b97c-3099-4d20-b35c-7b828db8d889\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-g5j4t"
Jul 22 02:28:49.007260 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:49.007239    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:49.010117 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:49.010096    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjkrw\" (UniqueName: \"kubernetes.io/projected/40c1b97c-3099-4d20-b35c-7b828db8d889-kube-api-access-xjkrw\") pod \"limitador-limitador-64c94fd64f-g5j4t\" (UID: \"40c1b97c-3099-4d20-b35c-7b828db8d889\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-g5j4t"
Jul 22 02:28:49.119049 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:49.119019    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-g5j4t"
Jul 22 02:28:49.245890 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:49.245867    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-g5j4t"]
Jul 22 02:28:49.248179 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:28:49.248149    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c1b97c_3099_4d20_b35c_7b828db8d889.slice/crio-dffdc2e020a02c4e43baca75a91903411712d65bbac76f9fb03ab3d6e5384bcd WatchSource:0}: Error finding container dffdc2e020a02c4e43baca75a91903411712d65bbac76f9fb03ab3d6e5384bcd: Status 404 returned error can't find the container with id dffdc2e020a02c4e43baca75a91903411712d65bbac76f9fb03ab3d6e5384bcd
Jul 22 02:28:49.605119 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:49.605077    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-zd6p4"]
Jul 22 02:28:49.607652 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:49.607633    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-zd6p4"
Jul 22 02:28:49.611256 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:49.611235    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4v575\""
Jul 22 02:28:49.618153 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:49.618132    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-zd6p4"]
Jul 22 02:28:49.656267 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:49.656230    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:49.704366 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:49.704329    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrkfr\" (UniqueName: \"kubernetes.io/projected/bdd2023f-821a-47f0-a10e-9bc725260770-kube-api-access-zrkfr\") pod \"authorino-58cd955675-zd6p4\" (UID: \"bdd2023f-821a-47f0-a10e-9bc725260770\") " pod="kuadrant-system/authorino-58cd955675-zd6p4"
Jul 22 02:28:49.757507 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:49.757477    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:49.760511 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:49.760435    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-g5j4t" event={"ID":"40c1b97c-3099-4d20-b35c-7b828db8d889","Type":"ContainerStarted","Data":"dffdc2e020a02c4e43baca75a91903411712d65bbac76f9fb03ab3d6e5384bcd"}
Jul 22 02:28:49.804869 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:49.804818    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrkfr\" (UniqueName: \"kubernetes.io/projected/bdd2023f-821a-47f0-a10e-9bc725260770-kube-api-access-zrkfr\") pod \"authorino-58cd955675-zd6p4\" (UID: \"bdd2023f-821a-47f0-a10e-9bc725260770\") " pod="kuadrant-system/authorino-58cd955675-zd6p4"
Jul 22 02:28:49.821597 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:49.821566    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrkfr\" (UniqueName: \"kubernetes.io/projected/bdd2023f-821a-47f0-a10e-9bc725260770-kube-api-access-zrkfr\") pod \"authorino-58cd955675-zd6p4\" (UID: \"bdd2023f-821a-47f0-a10e-9bc725260770\") " pod="kuadrant-system/authorino-58cd955675-zd6p4"
Jul 22 02:28:49.918555 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:49.918456    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-zd6p4"
Jul 22 02:28:50.079838 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:50.079722    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-zd6p4"]
Jul 22 02:28:50.082636 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:28:50.082603    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdd2023f_821a_47f0_a10e_9bc725260770.slice/crio-4b939eddaabe49d4d03c7110c5dd348afda13d47c546d735c8e3fd0d61568f29 WatchSource:0}: Error finding container 4b939eddaabe49d4d03c7110c5dd348afda13d47c546d735c8e3fd0d61568f29: Status 404 returned error can't find the container with id 4b939eddaabe49d4d03c7110c5dd348afda13d47c546d735c8e3fd0d61568f29
Jul 22 02:28:50.772451 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:50.772410    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-zd6p4" event={"ID":"bdd2023f-821a-47f0-a10e-9bc725260770","Type":"ContainerStarted","Data":"4b939eddaabe49d4d03c7110c5dd348afda13d47c546d735c8e3fd0d61568f29"}
Jul 22 02:28:52.784356 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:52.784323    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-g5j4t" event={"ID":"40c1b97c-3099-4d20-b35c-7b828db8d889","Type":"ContainerStarted","Data":"d40b6c699d295031b5bccfe57c9d22d8c12fb02164a3570338fa3b6cfab4c340"}
Jul 22 02:28:52.784735 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:52.784429    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-g5j4t"
Jul 22 02:28:52.804579 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:52.804530    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-g5j4t" podStartSLOduration=2.186682787 podStartE2EDuration="4.804511707s" totalImagesPullingTime="2.61782892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:52.80254245 +0000 UTC m=+483.650879025" watchObservedRunningTime="2026-07-22 02:28:52.804511707 +0000 UTC m=+483.652848280"
Jul 22 02:28:53.732191 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:53.732163    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:53.789741 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:53.789701    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-zd6p4" event={"ID":"bdd2023f-821a-47f0-a10e-9bc725260770","Type":"ContainerStarted","Data":"e20bde7ae841aee71a092306287b50f304083ae173d1f2aa11a296253c39e0b1"}
Jul 22 02:28:53.806600 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:53.806549    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-zd6p4" podStartSLOduration=1.24888794 podStartE2EDuration="4.806533254s" totalImagesPullingTime="3.557645314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:53.804407433 +0000 UTC m=+484.652744008" watchObservedRunningTime="2026-07-22 02:28:53.806533254 +0000 UTC m=+484.654869828"
Jul 22 02:28:53.816918 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:53.816890    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:54.504527 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:54.504489    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:54.610614 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:54.610580    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:54.738339 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:54.738313    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:55.173932 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:55.173892    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-zd6p4"]
Jul 22 02:28:55.239588 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:55.239553    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:55.340238 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:55.340205    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:55.484282 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:55.484210    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:28:55.797764 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:55.797671    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-zd6p4" podUID="bdd2023f-821a-47f0-a10e-9bc725260770" containerName="authorino" containerID="cri-o://e20bde7ae841aee71a092306287b50f304083ae173d1f2aa11a296253c39e0b1" gracePeriod=30
Jul 22 02:28:56.038082 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:56.038056    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-zd6p4"
Jul 22 02:28:56.056514 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:56.056437    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrkfr\" (UniqueName: \"kubernetes.io/projected/bdd2023f-821a-47f0-a10e-9bc725260770-kube-api-access-zrkfr\") pod \"bdd2023f-821a-47f0-a10e-9bc725260770\" (UID: \"bdd2023f-821a-47f0-a10e-9bc725260770\") "
Jul 22 02:28:56.058601 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:56.058568    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd2023f-821a-47f0-a10e-9bc725260770-kube-api-access-zrkfr" (OuterVolumeSpecName: "kube-api-access-zrkfr") pod "bdd2023f-821a-47f0-a10e-9bc725260770" (UID: "bdd2023f-821a-47f0-a10e-9bc725260770"). InnerVolumeSpecName "kube-api-access-zrkfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:28:56.157733 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:56.157695    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrkfr\" (UniqueName: \"kubernetes.io/projected/bdd2023f-821a-47f0-a10e-9bc725260770-kube-api-access-zrkfr\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:28:56.808529 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:56.808495    2565 generic.go:356] "Generic (PLEG): container finished" podID="bdd2023f-821a-47f0-a10e-9bc725260770" containerID="e20bde7ae841aee71a092306287b50f304083ae173d1f2aa11a296253c39e0b1" exitCode=0
Jul 22 02:28:56.808935 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:56.808542    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-zd6p4"
Jul 22 02:28:56.808935 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:56.808573    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-zd6p4" event={"ID":"bdd2023f-821a-47f0-a10e-9bc725260770","Type":"ContainerDied","Data":"e20bde7ae841aee71a092306287b50f304083ae173d1f2aa11a296253c39e0b1"}
Jul 22 02:28:56.808935 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:56.808618    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-zd6p4" event={"ID":"bdd2023f-821a-47f0-a10e-9bc725260770","Type":"ContainerDied","Data":"4b939eddaabe49d4d03c7110c5dd348afda13d47c546d735c8e3fd0d61568f29"}
Jul 22 02:28:56.808935 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:56.808639    2565 scope.go:117] "RemoveContainer" containerID="e20bde7ae841aee71a092306287b50f304083ae173d1f2aa11a296253c39e0b1"
Jul 22 02:28:56.817312 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:56.817293    2565 scope.go:117] "RemoveContainer" containerID="e20bde7ae841aee71a092306287b50f304083ae173d1f2aa11a296253c39e0b1"
Jul 22 02:28:56.817574 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:28:56.817557    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e20bde7ae841aee71a092306287b50f304083ae173d1f2aa11a296253c39e0b1\": container with ID starting with e20bde7ae841aee71a092306287b50f304083ae173d1f2aa11a296253c39e0b1 not found: ID does not exist" containerID="e20bde7ae841aee71a092306287b50f304083ae173d1f2aa11a296253c39e0b1"
Jul 22 02:28:56.817615 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:56.817583    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20bde7ae841aee71a092306287b50f304083ae173d1f2aa11a296253c39e0b1"} err="failed to get container status \"e20bde7ae841aee71a092306287b50f304083ae173d1f2aa11a296253c39e0b1\": rpc error: code = NotFound desc = could not find container \"e20bde7ae841aee71a092306287b50f304083ae173d1f2aa11a296253c39e0b1\": container with ID starting with e20bde7ae841aee71a092306287b50f304083ae173d1f2aa11a296253c39e0b1 not found: ID does not exist"
Jul 22 02:28:56.829859 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:56.829822    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-zd6p4"]
Jul 22 02:28:56.834172 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:56.834148    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-zd6p4"]
Jul 22 02:28:57.725546 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:28:57.725516    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdd2023f-821a-47f0-a10e-9bc725260770" path="/var/lib/kubelet/pods/bdd2023f-821a-47f0-a10e-9bc725260770/volumes"
Jul 22 02:29:03.791236 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:03.791210    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-g5j4t"
Jul 22 02:29:05.284940 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:05.284910    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:29:05.388084 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:05.388044    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:29:05.488282 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:05.488240    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:29:06.167870 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:06.167819    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-g5j4t"]
Jul 22 02:29:06.168059 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:06.168024    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-g5j4t" podUID="40c1b97c-3099-4d20-b35c-7b828db8d889" containerName="limitador" containerID="cri-o://d40b6c699d295031b5bccfe57c9d22d8c12fb02164a3570338fa3b6cfab4c340" gracePeriod=30
Jul 22 02:29:06.723798 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:06.723773    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-g5j4t"
Jul 22 02:29:06.741965 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:06.741890    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40c1b97c-3099-4d20-b35c-7b828db8d889-config-file\") pod \"40c1b97c-3099-4d20-b35c-7b828db8d889\" (UID: \"40c1b97c-3099-4d20-b35c-7b828db8d889\") "
Jul 22 02:29:06.741965 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:06.741940    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjkrw\" (UniqueName: \"kubernetes.io/projected/40c1b97c-3099-4d20-b35c-7b828db8d889-kube-api-access-xjkrw\") pod \"40c1b97c-3099-4d20-b35c-7b828db8d889\" (UID: \"40c1b97c-3099-4d20-b35c-7b828db8d889\") "
Jul 22 02:29:06.742313 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:06.742290    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c1b97c-3099-4d20-b35c-7b828db8d889-config-file" (OuterVolumeSpecName: "config-file") pod "40c1b97c-3099-4d20-b35c-7b828db8d889" (UID: "40c1b97c-3099-4d20-b35c-7b828db8d889"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:29:06.744106 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:06.744076    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c1b97c-3099-4d20-b35c-7b828db8d889-kube-api-access-xjkrw" (OuterVolumeSpecName: "kube-api-access-xjkrw") pod "40c1b97c-3099-4d20-b35c-7b828db8d889" (UID: "40c1b97c-3099-4d20-b35c-7b828db8d889"). InnerVolumeSpecName "kube-api-access-xjkrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:29:06.843372 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:06.843336    2565 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40c1b97c-3099-4d20-b35c-7b828db8d889-config-file\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:29:06.843372 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:06.843366    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjkrw\" (UniqueName: \"kubernetes.io/projected/40c1b97c-3099-4d20-b35c-7b828db8d889-kube-api-access-xjkrw\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:29:06.853779 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:06.853741    2565 generic.go:356] "Generic (PLEG): container finished" podID="40c1b97c-3099-4d20-b35c-7b828db8d889" containerID="d40b6c699d295031b5bccfe57c9d22d8c12fb02164a3570338fa3b6cfab4c340" exitCode=0
Jul 22 02:29:06.853957 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:06.853816    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-g5j4t"
Jul 22 02:29:06.853957 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:06.853830    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-g5j4t" event={"ID":"40c1b97c-3099-4d20-b35c-7b828db8d889","Type":"ContainerDied","Data":"d40b6c699d295031b5bccfe57c9d22d8c12fb02164a3570338fa3b6cfab4c340"}
Jul 22 02:29:06.853957 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:06.853916    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-g5j4t" event={"ID":"40c1b97c-3099-4d20-b35c-7b828db8d889","Type":"ContainerDied","Data":"dffdc2e020a02c4e43baca75a91903411712d65bbac76f9fb03ab3d6e5384bcd"}
Jul 22 02:29:06.853957 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:06.853947    2565 scope.go:117] "RemoveContainer" containerID="d40b6c699d295031b5bccfe57c9d22d8c12fb02164a3570338fa3b6cfab4c340"
Jul 22 02:29:06.862683 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:06.862668    2565 scope.go:117] "RemoveContainer" containerID="d40b6c699d295031b5bccfe57c9d22d8c12fb02164a3570338fa3b6cfab4c340"
Jul 22 02:29:06.862991 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:29:06.862968    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d40b6c699d295031b5bccfe57c9d22d8c12fb02164a3570338fa3b6cfab4c340\": container with ID starting with d40b6c699d295031b5bccfe57c9d22d8c12fb02164a3570338fa3b6cfab4c340 not found: ID does not exist" containerID="d40b6c699d295031b5bccfe57c9d22d8c12fb02164a3570338fa3b6cfab4c340"
Jul 22 02:29:06.863081 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:06.862998    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d40b6c699d295031b5bccfe57c9d22d8c12fb02164a3570338fa3b6cfab4c340"} err="failed to get container status \"d40b6c699d295031b5bccfe57c9d22d8c12fb02164a3570338fa3b6cfab4c340\": rpc error: code = NotFound desc = could not find container \"d40b6c699d295031b5bccfe57c9d22d8c12fb02164a3570338fa3b6cfab4c340\": container with ID starting with d40b6c699d295031b5bccfe57c9d22d8c12fb02164a3570338fa3b6cfab4c340 not found: ID does not exist"
Jul 22 02:29:06.875707 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:06.875686    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-g5j4t"]
Jul 22 02:29:06.879530 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:06.879510    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-g5j4t"]
Jul 22 02:29:07.725807 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:07.725773    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40c1b97c-3099-4d20-b35c-7b828db8d889" path="/var/lib/kubelet/pods/40c1b97c-3099-4d20-b35c-7b828db8d889/volumes"
Jul 22 02:29:10.040463 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.040433    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:29:10.132039 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.132008    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:29:10.447868 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.447825    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-njx2t"]
Jul 22 02:29:10.448226 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.448208    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c1b97c-3099-4d20-b35c-7b828db8d889" containerName="limitador"
Jul 22 02:29:10.448310 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.448228    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c1b97c-3099-4d20-b35c-7b828db8d889" containerName="limitador"
Jul 22 02:29:10.448310 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.448258    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdd2023f-821a-47f0-a10e-9bc725260770" containerName="authorino"
Jul 22 02:29:10.448310 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.448267    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdd2023f-821a-47f0-a10e-9bc725260770" containerName="authorino"
Jul 22 02:29:10.448452 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.448355    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="40c1b97c-3099-4d20-b35c-7b828db8d889" containerName="limitador"
Jul 22 02:29:10.448452 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.448367    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdd2023f-821a-47f0-a10e-9bc725260770" containerName="authorino"
Jul 22 02:29:10.450976 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.450956    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-njx2t"
Jul 22 02:29:10.459880 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.457380    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 22 02:29:10.459880 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.457673    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-2hw6n\""
Jul 22 02:29:10.459880 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.458274    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 22 02:29:10.459880 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.458533    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 22 02:29:10.466286 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.466258    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-njx2t"]
Jul 22 02:29:10.471648 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.471626    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5caa01c0-91bb-4b72-877e-27e01d39395c-data\") pod \"postgres-69fb4594cf-njx2t\" (UID: \"5caa01c0-91bb-4b72-877e-27e01d39395c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-njx2t"
Jul 22 02:29:10.471738 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.471658    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n6g6\" (UniqueName: \"kubernetes.io/projected/5caa01c0-91bb-4b72-877e-27e01d39395c-kube-api-access-2n6g6\") pod \"postgres-69fb4594cf-njx2t\" (UID: \"5caa01c0-91bb-4b72-877e-27e01d39395c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-njx2t"
Jul 22 02:29:10.487855 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.487832    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:29:10.572372 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.572342    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5caa01c0-91bb-4b72-877e-27e01d39395c-data\") pod \"postgres-69fb4594cf-njx2t\" (UID: \"5caa01c0-91bb-4b72-877e-27e01d39395c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-njx2t"
Jul 22 02:29:10.572505 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.572376    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n6g6\" (UniqueName: \"kubernetes.io/projected/5caa01c0-91bb-4b72-877e-27e01d39395c-kube-api-access-2n6g6\") pod \"postgres-69fb4594cf-njx2t\" (UID: \"5caa01c0-91bb-4b72-877e-27e01d39395c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-njx2t"
Jul 22 02:29:10.572704 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.572683    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5caa01c0-91bb-4b72-877e-27e01d39395c-data\") pod \"postgres-69fb4594cf-njx2t\" (UID: \"5caa01c0-91bb-4b72-877e-27e01d39395c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-njx2t"
Jul 22 02:29:10.581029 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.581010    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n6g6\" (UniqueName: \"kubernetes.io/projected/5caa01c0-91bb-4b72-877e-27e01d39395c-kube-api-access-2n6g6\") pod \"postgres-69fb4594cf-njx2t\" (UID: \"5caa01c0-91bb-4b72-877e-27e01d39395c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-njx2t"
Jul 22 02:29:10.768404 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.768329    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-njx2t"
Jul 22 02:29:10.900518 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:10.900490    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-njx2t"]
Jul 22 02:29:10.902984 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:29:10.902958    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5caa01c0_91bb_4b72_877e_27e01d39395c.slice/crio-b4136d263ff22fa0e94b8edc331fc110a23310b1fee0910212d70bf75953d52c WatchSource:0}: Error finding container b4136d263ff22fa0e94b8edc331fc110a23310b1fee0910212d70bf75953d52c: Status 404 returned error can't find the container with id b4136d263ff22fa0e94b8edc331fc110a23310b1fee0910212d70bf75953d52c
Jul 22 02:29:11.879937 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:11.879844    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-njx2t" event={"ID":"5caa01c0-91bb-4b72-877e-27e01d39395c","Type":"ContainerStarted","Data":"b4136d263ff22fa0e94b8edc331fc110a23310b1fee0910212d70bf75953d52c"}
Jul 22 02:29:15.306868 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:15.306822    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:29:15.360033 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:15.360001    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:29:26.944401 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:26.944370    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-njx2t" event={"ID":"5caa01c0-91bb-4b72-877e-27e01d39395c","Type":"ContainerStarted","Data":"359944e33744671325e92eea4149da57b9c40035bea003243ef9e6a810c5e6d4"}
Jul 22 02:29:26.944772 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:26.944499    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-njx2t"
Jul 22 02:29:26.962983 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:26.962933    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-njx2t" podStartSLOduration=1.083493738 podStartE2EDuration="16.962919748s" totalImagesPullingTime="15.87942601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:26.961254919 +0000 UTC m=+517.809591497" watchObservedRunningTime="2026-07-22 02:29:26.962919748 +0000 UTC m=+517.811256321"
Jul 22 02:29:32.978153 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:32.978120    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-njx2t"
Jul 22 02:29:33.293476 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:33.293397    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:29:33.532609 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:33.532569    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gb27k"]
Jul 22 02:29:33.535903 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:33.535879    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-gb27k"
Jul 22 02:29:33.538314 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:33.538291    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4v575\""
Jul 22 02:29:33.543345 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:33.543254    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gb27k"]
Jul 22 02:29:33.662011 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:33.661984    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thvfv\" (UniqueName: \"kubernetes.io/projected/c436a633-0265-47f3-9427-ca2eb0f836ea-kube-api-access-thvfv\") pod \"authorino-5d4b67c446-gb27k\" (UID: \"c436a633-0265-47f3-9427-ca2eb0f836ea\") " pod="kuadrant-system/authorino-5d4b67c446-gb27k"
Jul 22 02:29:33.673551 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:33.673500    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:29:33.762916 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:33.762885    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thvfv\" (UniqueName: \"kubernetes.io/projected/c436a633-0265-47f3-9427-ca2eb0f836ea-kube-api-access-thvfv\") pod \"authorino-5d4b67c446-gb27k\" (UID: \"c436a633-0265-47f3-9427-ca2eb0f836ea\") " pod="kuadrant-system/authorino-5d4b67c446-gb27k"
Jul 22 02:29:33.772209 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:33.772186    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thvfv\" (UniqueName: \"kubernetes.io/projected/c436a633-0265-47f3-9427-ca2eb0f836ea-kube-api-access-thvfv\") pod \"authorino-5d4b67c446-gb27k\" (UID: \"c436a633-0265-47f3-9427-ca2eb0f836ea\") " pod="kuadrant-system/authorino-5d4b67c446-gb27k"
Jul 22 02:29:33.773414 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:33.773394    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gb27k"]
Jul 22 02:29:33.773600 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:33.773588    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-gb27k"
Jul 22 02:29:33.913550 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:33.913422    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-866d947867-vftd7"]
Jul 22 02:29:33.918186 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:29:33.918154    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc436a633_0265_47f3_9427_ca2eb0f836ea.slice/crio-cb53e169a9b531cd254fa858cf8175fe70bba5796881a9f0ff56a6a1dddf880a WatchSource:0}: Error finding container cb53e169a9b531cd254fa858cf8175fe70bba5796881a9f0ff56a6a1dddf880a: Status 404 returned error can't find the container with id cb53e169a9b531cd254fa858cf8175fe70bba5796881a9f0ff56a6a1dddf880a
Jul 22 02:29:33.920587 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:33.920388    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gb27k"]
Jul 22 02:29:33.920587 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:33.920421    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-866d947867-vftd7"]
Jul 22 02:29:33.920587 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:33.920499    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-866d947867-vftd7"
Jul 22 02:29:33.922661 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:33.922643    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 22 02:29:33.977458 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:33.977430    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-gb27k" event={"ID":"c436a633-0265-47f3-9427-ca2eb0f836ea","Type":"ContainerStarted","Data":"cb53e169a9b531cd254fa858cf8175fe70bba5796881a9f0ff56a6a1dddf880a"}
Jul 22 02:29:34.065050 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:34.065022    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8lvw\" (UniqueName: \"kubernetes.io/projected/d141a5c2-91b4-4fff-833d-ef677140bcd1-kube-api-access-n8lvw\") pod \"authorino-866d947867-vftd7\" (UID: \"d141a5c2-91b4-4fff-833d-ef677140bcd1\") " pod="kuadrant-system/authorino-866d947867-vftd7"
Jul 22 02:29:34.065340 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:34.065074    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d141a5c2-91b4-4fff-833d-ef677140bcd1-tls-cert\") pod \"authorino-866d947867-vftd7\" (UID: \"d141a5c2-91b4-4fff-833d-ef677140bcd1\") " pod="kuadrant-system/authorino-866d947867-vftd7"
Jul 22 02:29:34.165652 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:34.165586    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d141a5c2-91b4-4fff-833d-ef677140bcd1-tls-cert\") pod \"authorino-866d947867-vftd7\" (UID: \"d141a5c2-91b4-4fff-833d-ef677140bcd1\") " pod="kuadrant-system/authorino-866d947867-vftd7"
Jul 22 02:29:34.165652 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:34.165640    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8lvw\" (UniqueName: \"kubernetes.io/projected/d141a5c2-91b4-4fff-833d-ef677140bcd1-kube-api-access-n8lvw\") pod \"authorino-866d947867-vftd7\" (UID: \"d141a5c2-91b4-4fff-833d-ef677140bcd1\") " pod="kuadrant-system/authorino-866d947867-vftd7"
Jul 22 02:29:34.167955 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:34.167925    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d141a5c2-91b4-4fff-833d-ef677140bcd1-tls-cert\") pod \"authorino-866d947867-vftd7\" (UID: \"d141a5c2-91b4-4fff-833d-ef677140bcd1\") " pod="kuadrant-system/authorino-866d947867-vftd7"
Jul 22 02:29:34.173393 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:34.173373    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8lvw\" (UniqueName: \"kubernetes.io/projected/d141a5c2-91b4-4fff-833d-ef677140bcd1-kube-api-access-n8lvw\") pod \"authorino-866d947867-vftd7\" (UID: \"d141a5c2-91b4-4fff-833d-ef677140bcd1\") " pod="kuadrant-system/authorino-866d947867-vftd7"
Jul 22 02:29:34.233214 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:34.233192    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-866d947867-vftd7"
Jul 22 02:29:34.367316 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:34.366902    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-866d947867-vftd7"]
Jul 22 02:29:34.369890 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:29:34.369865    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd141a5c2_91b4_4fff_833d_ef677140bcd1.slice/crio-63786980a60a42f9516d652bfc70e1f56fa814512d125a247880b6528c1f721f WatchSource:0}: Error finding container 63786980a60a42f9516d652bfc70e1f56fa814512d125a247880b6528c1f721f: Status 404 returned error can't find the container with id 63786980a60a42f9516d652bfc70e1f56fa814512d125a247880b6528c1f721f
Jul 22 02:29:34.984108 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:34.984009    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-866d947867-vftd7" event={"ID":"d141a5c2-91b4-4fff-833d-ef677140bcd1","Type":"ContainerStarted","Data":"55e22bb7334ef19c2a1f00b4f341da23ed14f71917a56ab26cdd0d2d6a6ee213"}
Jul 22 02:29:34.984108 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:34.984049    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-866d947867-vftd7" event={"ID":"d141a5c2-91b4-4fff-833d-ef677140bcd1","Type":"ContainerStarted","Data":"63786980a60a42f9516d652bfc70e1f56fa814512d125a247880b6528c1f721f"}
Jul 22 02:29:34.985380 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:34.985352    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-gb27k" event={"ID":"c436a633-0265-47f3-9427-ca2eb0f836ea","Type":"ContainerStarted","Data":"42faa07a4be3779fafca774e306e8f4dcfc25cea5b77a4e3da6484230490c666"}
Jul 22 02:29:34.985483 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:34.985414    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-gb27k" podUID="c436a633-0265-47f3-9427-ca2eb0f836ea" containerName="authorino" containerID="cri-o://42faa07a4be3779fafca774e306e8f4dcfc25cea5b77a4e3da6484230490c666" gracePeriod=30
Jul 22 02:29:34.998343 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:34.998294    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-866d947867-vftd7" podStartSLOduration=1.6954507909999998 podStartE2EDuration="1.998277733s" totalImagesPullingTime="302.826942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:34.99822047 +0000 UTC m=+525.846557045" watchObservedRunningTime="2026-07-22 02:29:34.998277733 +0000 UTC m=+525.846614308"
Jul 22 02:29:35.013496 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:35.013446    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-gb27k" podStartSLOduration=1.713612525 podStartE2EDuration="2.01343166s" totalImagesPullingTime="299.819135ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:35.011886153 +0000 UTC m=+525.860222731" watchObservedRunningTime="2026-07-22 02:29:35.01343166 +0000 UTC m=+525.861768233"
Jul 22 02:29:35.225344 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:35.225320    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-gb27k"
Jul 22 02:29:35.375900 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:35.375869    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thvfv\" (UniqueName: \"kubernetes.io/projected/c436a633-0265-47f3-9427-ca2eb0f836ea-kube-api-access-thvfv\") pod \"c436a633-0265-47f3-9427-ca2eb0f836ea\" (UID: \"c436a633-0265-47f3-9427-ca2eb0f836ea\") "
Jul 22 02:29:35.377829 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:35.377799    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c436a633-0265-47f3-9427-ca2eb0f836ea-kube-api-access-thvfv" (OuterVolumeSpecName: "kube-api-access-thvfv") pod "c436a633-0265-47f3-9427-ca2eb0f836ea" (UID: "c436a633-0265-47f3-9427-ca2eb0f836ea"). InnerVolumeSpecName "kube-api-access-thvfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:29:35.476989 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:35.476962    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thvfv\" (UniqueName: \"kubernetes.io/projected/c436a633-0265-47f3-9427-ca2eb0f836ea-kube-api-access-thvfv\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:29:35.990241 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:35.990156    2565 generic.go:356] "Generic (PLEG): container finished" podID="c436a633-0265-47f3-9427-ca2eb0f836ea" containerID="42faa07a4be3779fafca774e306e8f4dcfc25cea5b77a4e3da6484230490c666" exitCode=0
Jul 22 02:29:35.990241 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:35.990195    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-gb27k"
Jul 22 02:29:35.990471 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:35.990237    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-gb27k" event={"ID":"c436a633-0265-47f3-9427-ca2eb0f836ea","Type":"ContainerDied","Data":"42faa07a4be3779fafca774e306e8f4dcfc25cea5b77a4e3da6484230490c666"}
Jul 22 02:29:35.990471 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:35.990276    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-gb27k" event={"ID":"c436a633-0265-47f3-9427-ca2eb0f836ea","Type":"ContainerDied","Data":"cb53e169a9b531cd254fa858cf8175fe70bba5796881a9f0ff56a6a1dddf880a"}
Jul 22 02:29:35.990471 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:35.990293    2565 scope.go:117] "RemoveContainer" containerID="42faa07a4be3779fafca774e306e8f4dcfc25cea5b77a4e3da6484230490c666"
Jul 22 02:29:35.998649 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:35.998634    2565 scope.go:117] "RemoveContainer" containerID="42faa07a4be3779fafca774e306e8f4dcfc25cea5b77a4e3da6484230490c666"
Jul 22 02:29:35.998915 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:29:35.998898    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42faa07a4be3779fafca774e306e8f4dcfc25cea5b77a4e3da6484230490c666\": container with ID starting with 42faa07a4be3779fafca774e306e8f4dcfc25cea5b77a4e3da6484230490c666 not found: ID does not exist" containerID="42faa07a4be3779fafca774e306e8f4dcfc25cea5b77a4e3da6484230490c666"
Jul 22 02:29:35.998968 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:35.998922    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42faa07a4be3779fafca774e306e8f4dcfc25cea5b77a4e3da6484230490c666"} err="failed to get container status \"42faa07a4be3779fafca774e306e8f4dcfc25cea5b77a4e3da6484230490c666\": rpc error: code = NotFound desc = could not find container \"42faa07a4be3779fafca774e306e8f4dcfc25cea5b77a4e3da6484230490c666\": container with ID starting with 42faa07a4be3779fafca774e306e8f4dcfc25cea5b77a4e3da6484230490c666 not found: ID does not exist"
Jul 22 02:29:36.005998 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:36.005977    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gb27k"]
Jul 22 02:29:36.009963 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:36.009944    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gb27k"]
Jul 22 02:29:37.725590 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:37.725560    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c436a633-0265-47f3-9427-ca2eb0f836ea" path="/var/lib/kubelet/pods/c436a633-0265-47f3-9427-ca2eb0f836ea/volumes"
Jul 22 02:29:39.681353 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:39.681314    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:29:39.717389 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:39.717351    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-69bd6c5658-v5lxg"]
Jul 22 02:29:39.717723 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:39.717711    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c436a633-0265-47f3-9427-ca2eb0f836ea" containerName="authorino"
Jul 22 02:29:39.717808 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:39.717725    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c436a633-0265-47f3-9427-ca2eb0f836ea" containerName="authorino"
Jul 22 02:29:39.717808 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:39.717779    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c436a633-0265-47f3-9427-ca2eb0f836ea" containerName="authorino"
Jul 22 02:29:39.721972 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:39.721944    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69bd6c5658-v5lxg"
Jul 22 02:29:39.724539 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:39.724505    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 22 02:29:39.724539 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:39.724538    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rddtn\""
Jul 22 02:29:39.724787 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:39.724516    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 22 02:29:39.735839 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:39.735810    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69bd6c5658-v5lxg"]
Jul 22 02:29:39.779142 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:39.779106    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:29:39.814291 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:39.814263    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmw7z\" (UniqueName: \"kubernetes.io/projected/0c5c9291-4268-45df-84fc-cb663d022897-kube-api-access-rmw7z\") pod \"maas-controller-69bd6c5658-v5lxg\" (UID: \"0c5c9291-4268-45df-84fc-cb663d022897\") " pod="opendatahub/maas-controller-69bd6c5658-v5lxg"
Jul 22 02:29:39.814419 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:39.814312    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c5c9291-4268-45df-84fc-cb663d022897-webhook-cert\") pod \"maas-controller-69bd6c5658-v5lxg\" (UID: \"0c5c9291-4268-45df-84fc-cb663d022897\") " pod="opendatahub/maas-controller-69bd6c5658-v5lxg"
Jul 22 02:29:39.845988 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:39.845962    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:29:39.915665 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:39.915633    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c5c9291-4268-45df-84fc-cb663d022897-webhook-cert\") pod \"maas-controller-69bd6c5658-v5lxg\" (UID: \"0c5c9291-4268-45df-84fc-cb663d022897\") " pod="opendatahub/maas-controller-69bd6c5658-v5lxg"
Jul 22 02:29:39.915781 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:39.915724    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmw7z\" (UniqueName: \"kubernetes.io/projected/0c5c9291-4268-45df-84fc-cb663d022897-kube-api-access-rmw7z\") pod \"maas-controller-69bd6c5658-v5lxg\" (UID: \"0c5c9291-4268-45df-84fc-cb663d022897\") " pod="opendatahub/maas-controller-69bd6c5658-v5lxg"
Jul 22 02:29:39.915781 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:29:39.915759    2565 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 22 02:29:39.915868 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:29:39.915834    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c5c9291-4268-45df-84fc-cb663d022897-webhook-cert podName:0c5c9291-4268-45df-84fc-cb663d022897 nodeName:}" failed. No retries permitted until 2026-07-22 02:29:40.415817503 +0000 UTC m=+531.264154065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/0c5c9291-4268-45df-84fc-cb663d022897-webhook-cert") pod "maas-controller-69bd6c5658-v5lxg" (UID: "0c5c9291-4268-45df-84fc-cb663d022897") : secret "maas-controller-webhook-cert" not found
Jul 22 02:29:39.926578 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:39.926547    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmw7z\" (UniqueName: \"kubernetes.io/projected/0c5c9291-4268-45df-84fc-cb663d022897-kube-api-access-rmw7z\") pod \"maas-controller-69bd6c5658-v5lxg\" (UID: \"0c5c9291-4268-45df-84fc-cb663d022897\") " pod="opendatahub/maas-controller-69bd6c5658-v5lxg"
Jul 22 02:29:40.051682 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:40.051598    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:29:40.420373 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:40.420342    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c5c9291-4268-45df-84fc-cb663d022897-webhook-cert\") pod \"maas-controller-69bd6c5658-v5lxg\" (UID: \"0c5c9291-4268-45df-84fc-cb663d022897\") " pod="opendatahub/maas-controller-69bd6c5658-v5lxg"
Jul 22 02:29:40.422720 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:40.422701    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c5c9291-4268-45df-84fc-cb663d022897-webhook-cert\") pod \"maas-controller-69bd6c5658-v5lxg\" (UID: \"0c5c9291-4268-45df-84fc-cb663d022897\") " pod="opendatahub/maas-controller-69bd6c5658-v5lxg"
Jul 22 02:29:40.632682 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:40.632644    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69bd6c5658-v5lxg"
Jul 22 02:29:40.763383 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:40.763358    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69bd6c5658-v5lxg"]
Jul 22 02:29:40.765710 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:29:40.765681    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5c9291_4268_45df_84fc_cb663d022897.slice/crio-60e65fcfd9f76e4c855e59284c182cd5b99762211fedb0a0419302a49588462f WatchSource:0}: Error finding container 60e65fcfd9f76e4c855e59284c182cd5b99762211fedb0a0419302a49588462f: Status 404 returned error can't find the container with id 60e65fcfd9f76e4c855e59284c182cd5b99762211fedb0a0419302a49588462f
Jul 22 02:29:41.014440 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:41.014356    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69bd6c5658-v5lxg" event={"ID":"0c5c9291-4268-45df-84fc-cb663d022897","Type":"ContainerStarted","Data":"60e65fcfd9f76e4c855e59284c182cd5b99762211fedb0a0419302a49588462f"}
Jul 22 02:29:42.867115 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:42.867001    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:29:42.965493 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:42.965463    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:29:44.036888 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:44.036832    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69bd6c5658-v5lxg" event={"ID":"0c5c9291-4268-45df-84fc-cb663d022897","Type":"ContainerStarted","Data":"922effc54bfc8a3c5431b524a08fb116b3ce2037355ba05de0270014c0009d27"}
Jul 22 02:29:44.037336 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:44.036954    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-69bd6c5658-v5lxg"
Jul 22 02:29:44.058507 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:44.058462    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-69bd6c5658-v5lxg" podStartSLOduration=2.856906058 podStartE2EDuration="5.058449295s" totalImagesPullingTime="2.201543237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:44.056217998 +0000 UTC m=+534.904554582" watchObservedRunningTime="2026-07-22 02:29:44.058449295 +0000 UTC m=+534.906785868"
Jul 22 02:29:54.279952 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:54.279917    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69bd6c5658-v5lxg"]
Jul 22 02:29:54.280374 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:54.280097    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-69bd6c5658-v5lxg" podUID="0c5c9291-4268-45df-84fc-cb663d022897" containerName="manager" containerID="cri-o://922effc54bfc8a3c5431b524a08fb116b3ce2037355ba05de0270014c0009d27" gracePeriod=30
Jul 22 02:29:54.282193 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:54.282169    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-69bd6c5658-v5lxg"
Jul 22 02:29:54.515974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:54.514505    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69bd6c5658-v5lxg"
Jul 22 02:29:54.637884 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:54.637827    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmw7z\" (UniqueName: \"kubernetes.io/projected/0c5c9291-4268-45df-84fc-cb663d022897-kube-api-access-rmw7z\") pod \"0c5c9291-4268-45df-84fc-cb663d022897\" (UID: \"0c5c9291-4268-45df-84fc-cb663d022897\") "
Jul 22 02:29:54.638033 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:54.637887    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c5c9291-4268-45df-84fc-cb663d022897-webhook-cert\") pod \"0c5c9291-4268-45df-84fc-cb663d022897\" (UID: \"0c5c9291-4268-45df-84fc-cb663d022897\") "
Jul 22 02:29:54.640327 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:54.640292    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c5c9291-4268-45df-84fc-cb663d022897-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0c5c9291-4268-45df-84fc-cb663d022897" (UID: "0c5c9291-4268-45df-84fc-cb663d022897"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:29:54.640327 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:54.640306    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5c9291-4268-45df-84fc-cb663d022897-kube-api-access-rmw7z" (OuterVolumeSpecName: "kube-api-access-rmw7z") pod "0c5c9291-4268-45df-84fc-cb663d022897" (UID: "0c5c9291-4268-45df-84fc-cb663d022897"). InnerVolumeSpecName "kube-api-access-rmw7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:29:54.739521 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:54.739492    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmw7z\" (UniqueName: \"kubernetes.io/projected/0c5c9291-4268-45df-84fc-cb663d022897-kube-api-access-rmw7z\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:29:54.739521 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:54.739518    2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c5c9291-4268-45df-84fc-cb663d022897-webhook-cert\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:29:55.083675 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:55.083596    2565 generic.go:356] "Generic (PLEG): container finished" podID="0c5c9291-4268-45df-84fc-cb663d022897" containerID="922effc54bfc8a3c5431b524a08fb116b3ce2037355ba05de0270014c0009d27" exitCode=0
Jul 22 02:29:55.083675 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:55.083643    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69bd6c5658-v5lxg"
Jul 22 02:29:55.083844 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:55.083681    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69bd6c5658-v5lxg" event={"ID":"0c5c9291-4268-45df-84fc-cb663d022897","Type":"ContainerDied","Data":"922effc54bfc8a3c5431b524a08fb116b3ce2037355ba05de0270014c0009d27"}
Jul 22 02:29:55.083844 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:55.083738    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69bd6c5658-v5lxg" event={"ID":"0c5c9291-4268-45df-84fc-cb663d022897","Type":"ContainerDied","Data":"60e65fcfd9f76e4c855e59284c182cd5b99762211fedb0a0419302a49588462f"}
Jul 22 02:29:55.083844 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:55.083768    2565 scope.go:117] "RemoveContainer" containerID="922effc54bfc8a3c5431b524a08fb116b3ce2037355ba05de0270014c0009d27"
Jul 22 02:29:55.092772 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:55.092754    2565 scope.go:117] "RemoveContainer" containerID="922effc54bfc8a3c5431b524a08fb116b3ce2037355ba05de0270014c0009d27"
Jul 22 02:29:55.093072 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:29:55.093050    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922effc54bfc8a3c5431b524a08fb116b3ce2037355ba05de0270014c0009d27\": container with ID starting with 922effc54bfc8a3c5431b524a08fb116b3ce2037355ba05de0270014c0009d27 not found: ID does not exist" containerID="922effc54bfc8a3c5431b524a08fb116b3ce2037355ba05de0270014c0009d27"
Jul 22 02:29:55.093169 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:55.093077    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922effc54bfc8a3c5431b524a08fb116b3ce2037355ba05de0270014c0009d27"} err="failed to get container status \"922effc54bfc8a3c5431b524a08fb116b3ce2037355ba05de0270014c0009d27\": rpc error: code = NotFound desc = could not find container \"922effc54bfc8a3c5431b524a08fb116b3ce2037355ba05de0270014c0009d27\": container with ID starting with 922effc54bfc8a3c5431b524a08fb116b3ce2037355ba05de0270014c0009d27 not found: ID does not exist"
Jul 22 02:29:55.105009 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:55.104984    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69bd6c5658-v5lxg"]
Jul 22 02:29:55.109277 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:55.109255    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-69bd6c5658-v5lxg"]
Jul 22 02:29:55.726020 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:29:55.725989    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c5c9291-4268-45df-84fc-cb663d022897" path="/var/lib/kubelet/pods/0c5c9291-4268-45df-84fc-cb663d022897/volumes"
Jul 22 02:30:16.337038 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:16.336999    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:30:16.436474 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:16.436446    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:30:16.863182 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:16.863151    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:30:16.996133 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:16.996105    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-76d9db859c-rhpqn"]
Jul 22 02:30:16.996434 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:16.996422    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c5c9291-4268-45df-84fc-cb663d022897" containerName="manager"
Jul 22 02:30:16.996474 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:16.996435    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c5c9291-4268-45df-84fc-cb663d022897" containerName="manager"
Jul 22 02:30:16.996508 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:16.996498    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c5c9291-4268-45df-84fc-cb663d022897" containerName="manager"
Jul 22 02:30:17.000575 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.000555    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76d9db859c-rhpqn"
Jul 22 02:30:17.004270 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.004245    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b6vn4\""
Jul 22 02:30:17.005258 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.005239    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 22 02:30:17.025738 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.025710    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76d9db859c-rhpqn"]
Jul 22 02:30:17.042188 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.042163    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:30:17.116615 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.116535    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxx6z\" (UniqueName: \"kubernetes.io/projected/e393c1da-9ce4-4cf0-91a1-66ca299217d1-kube-api-access-lxx6z\") pod \"maas-api-76d9db859c-rhpqn\" (UID: \"e393c1da-9ce4-4cf0-91a1-66ca299217d1\") " pod="odh-ai-gateway-infra/maas-api-76d9db859c-rhpqn"
Jul 22 02:30:17.116615 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.116604    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e393c1da-9ce4-4cf0-91a1-66ca299217d1-maas-api-tls\") pod \"maas-api-76d9db859c-rhpqn\" (UID: \"e393c1da-9ce4-4cf0-91a1-66ca299217d1\") " pod="odh-ai-gateway-infra/maas-api-76d9db859c-rhpqn"
Jul 22 02:30:17.116800 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.116653    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e393c1da-9ce4-4cf0-91a1-66ca299217d1-openshift-service-ca\") pod \"maas-api-76d9db859c-rhpqn\" (UID: \"e393c1da-9ce4-4cf0-91a1-66ca299217d1\") " pod="odh-ai-gateway-infra/maas-api-76d9db859c-rhpqn"
Jul 22 02:30:17.218154 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.218106    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e393c1da-9ce4-4cf0-91a1-66ca299217d1-maas-api-tls\") pod \"maas-api-76d9db859c-rhpqn\" (UID: \"e393c1da-9ce4-4cf0-91a1-66ca299217d1\") " pod="odh-ai-gateway-infra/maas-api-76d9db859c-rhpqn"
Jul 22 02:30:17.218333 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.218181    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e393c1da-9ce4-4cf0-91a1-66ca299217d1-openshift-service-ca\") pod \"maas-api-76d9db859c-rhpqn\" (UID: \"e393c1da-9ce4-4cf0-91a1-66ca299217d1\") " pod="odh-ai-gateway-infra/maas-api-76d9db859c-rhpqn"
Jul 22 02:30:17.218333 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.218226    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxx6z\" (UniqueName: \"kubernetes.io/projected/e393c1da-9ce4-4cf0-91a1-66ca299217d1-kube-api-access-lxx6z\") pod \"maas-api-76d9db859c-rhpqn\" (UID: \"e393c1da-9ce4-4cf0-91a1-66ca299217d1\") " pod="odh-ai-gateway-infra/maas-api-76d9db859c-rhpqn"
Jul 22 02:30:17.218932 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.218912    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e393c1da-9ce4-4cf0-91a1-66ca299217d1-openshift-service-ca\") pod \"maas-api-76d9db859c-rhpqn\" (UID: \"e393c1da-9ce4-4cf0-91a1-66ca299217d1\") " pod="odh-ai-gateway-infra/maas-api-76d9db859c-rhpqn"
Jul 22 02:30:17.220838 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.220820    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e393c1da-9ce4-4cf0-91a1-66ca299217d1-maas-api-tls\") pod \"maas-api-76d9db859c-rhpqn\" (UID: \"e393c1da-9ce4-4cf0-91a1-66ca299217d1\") " pod="odh-ai-gateway-infra/maas-api-76d9db859c-rhpqn"
Jul 22 02:30:17.227696 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.227669    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxx6z\" (UniqueName: \"kubernetes.io/projected/e393c1da-9ce4-4cf0-91a1-66ca299217d1-kube-api-access-lxx6z\") pod \"maas-api-76d9db859c-rhpqn\" (UID: \"e393c1da-9ce4-4cf0-91a1-66ca299217d1\") " pod="odh-ai-gateway-infra/maas-api-76d9db859c-rhpqn"
Jul 22 02:30:17.310566 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.310529    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76d9db859c-rhpqn"
Jul 22 02:30:17.442465 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.442439    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:30:17.455865 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.455825    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76d9db859c-rhpqn"]
Jul 22 02:30:17.459313 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:30:17.459276    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode393c1da_9ce4_4cf0_91a1_66ca299217d1.slice/crio-9d71280805d2ec2ea433eee438de0c940ca9d6b2df7f69f4c604aaad786f8603 WatchSource:0}: Error finding container 9d71280805d2ec2ea433eee438de0c940ca9d6b2df7f69f4c604aaad786f8603: Status 404 returned error can't find the container with id 9d71280805d2ec2ea433eee438de0c940ca9d6b2df7f69f4c604aaad786f8603
Jul 22 02:30:17.541583 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.541549    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:30:17.580086 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.580044    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-vdmlh"]
Jul 22 02:30:17.582763 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.582741    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-vdmlh"
Jul 22 02:30:17.585442 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.585416    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 22 02:30:17.585562 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.585546    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bdbkt\""
Jul 22 02:30:17.596646 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.596622    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-vdmlh"]
Jul 22 02:30:17.622230 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.622198    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzk7s\" (UniqueName: \"kubernetes.io/projected/84cfc958-d425-471a-ab6f-cfcd0d8bc121-kube-api-access-gzk7s\") pod \"payload-processing-5bf47cd988-vdmlh\" (UID: \"84cfc958-d425-471a-ab6f-cfcd0d8bc121\") " pod="openshift-ingress/payload-processing-5bf47cd988-vdmlh"
Jul 22 02:30:17.622368 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.622249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84cfc958-d425-471a-ab6f-cfcd0d8bc121-plugins-config-volume\") pod \"payload-processing-5bf47cd988-vdmlh\" (UID: \"84cfc958-d425-471a-ab6f-cfcd0d8bc121\") " pod="openshift-ingress/payload-processing-5bf47cd988-vdmlh"
Jul 22 02:30:17.723034 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.722956    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzk7s\" (UniqueName: \"kubernetes.io/projected/84cfc958-d425-471a-ab6f-cfcd0d8bc121-kube-api-access-gzk7s\") pod \"payload-processing-5bf47cd988-vdmlh\" (UID: \"84cfc958-d425-471a-ab6f-cfcd0d8bc121\") " pod="openshift-ingress/payload-processing-5bf47cd988-vdmlh"
Jul 22 02:30:17.723034 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.723015    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84cfc958-d425-471a-ab6f-cfcd0d8bc121-plugins-config-volume\") pod \"payload-processing-5bf47cd988-vdmlh\" (UID: \"84cfc958-d425-471a-ab6f-cfcd0d8bc121\") " pod="openshift-ingress/payload-processing-5bf47cd988-vdmlh"
Jul 22 02:30:17.723749 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.723723    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84cfc958-d425-471a-ab6f-cfcd0d8bc121-plugins-config-volume\") pod \"payload-processing-5bf47cd988-vdmlh\" (UID: \"84cfc958-d425-471a-ab6f-cfcd0d8bc121\") " pod="openshift-ingress/payload-processing-5bf47cd988-vdmlh"
Jul 22 02:30:17.730901 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.730879    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzk7s\" (UniqueName: \"kubernetes.io/projected/84cfc958-d425-471a-ab6f-cfcd0d8bc121-kube-api-access-gzk7s\") pod \"payload-processing-5bf47cd988-vdmlh\" (UID: \"84cfc958-d425-471a-ab6f-cfcd0d8bc121\") " pod="openshift-ingress/payload-processing-5bf47cd988-vdmlh"
Jul 22 02:30:17.893091 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:17.893055    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-vdmlh"
Jul 22 02:30:18.034872 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:18.034787    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-vdmlh"]
Jul 22 02:30:18.036118 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:30:18.036091    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84cfc958_d425_471a_ab6f_cfcd0d8bc121.slice/crio-32eb7c76e8d1245d64b2f31e31c9cd58bc5da0ef5130c5ff7a0000d646a8aef6 WatchSource:0}: Error finding container 32eb7c76e8d1245d64b2f31e31c9cd58bc5da0ef5130c5ff7a0000d646a8aef6: Status 404 returned error can't find the container with id 32eb7c76e8d1245d64b2f31e31c9cd58bc5da0ef5130c5ff7a0000d646a8aef6
Jul 22 02:30:18.186633 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:18.186592    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-vdmlh" event={"ID":"84cfc958-d425-471a-ab6f-cfcd0d8bc121","Type":"ContainerStarted","Data":"32eb7c76e8d1245d64b2f31e31c9cd58bc5da0ef5130c5ff7a0000d646a8aef6"}
Jul 22 02:30:18.187926 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:18.187891    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76d9db859c-rhpqn" event={"ID":"e393c1da-9ce4-4cf0-91a1-66ca299217d1","Type":"ContainerStarted","Data":"9d71280805d2ec2ea433eee438de0c940ca9d6b2df7f69f4c604aaad786f8603"}
Jul 22 02:30:20.200491 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:20.200448    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76d9db859c-rhpqn" event={"ID":"e393c1da-9ce4-4cf0-91a1-66ca299217d1","Type":"ContainerStarted","Data":"445dff409cc64b2d9a792160927b1a96aeaf48ad025ddfa674687a7824fa8102"}
Jul 22 02:30:20.200964 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:20.200608    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-76d9db859c-rhpqn"
Jul 22 02:30:20.224397 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:20.223614    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-76d9db859c-rhpqn" podStartSLOduration=2.437102156 podStartE2EDuration="4.223592869s" totalImagesPullingTime="1.786490713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:30:20.221139017 +0000 UTC m=+571.069475592" watchObservedRunningTime="2026-07-22 02:30:20.223592869 +0000 UTC m=+571.071929446"
Jul 22 02:30:21.206556 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:21.206518    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-vdmlh" event={"ID":"84cfc958-d425-471a-ab6f-cfcd0d8bc121","Type":"ContainerStarted","Data":"00f956c659ae7c2bf1de693c1acf679585b01e9fd123428515156b992f403328"}
Jul 22 02:30:21.230202 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:21.230145    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-vdmlh" podStartSLOduration=1.426772704 podStartE2EDuration="4.230117723s" totalImagesPullingTime="2.803345019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:30:21.227627061 +0000 UTC m=+572.075963636" watchObservedRunningTime="2026-07-22 02:30:21.230117723 +0000 UTC m=+572.078454299"
Jul 22 02:30:22.211220 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:22.211188    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-vdmlh"
Jul 22 02:30:25.227151 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:25.227080    2565 generic.go:356] "Generic (PLEG): container finished" podID="3b40b006-204b-4291-901e-af512339c609" containerID="9e889e43809e710e3f28609253817be26d3572ec7eb9ce40129ecd6f4f6693ef" exitCode=0
Jul 22 02:30:25.227546 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:25.227160    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp" event={"ID":"3b40b006-204b-4291-901e-af512339c609","Type":"ContainerDied","Data":"9e889e43809e710e3f28609253817be26d3572ec7eb9ce40129ecd6f4f6693ef"}
Jul 22 02:30:25.227603 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:25.227549    2565 scope.go:117] "RemoveContainer" containerID="9e889e43809e710e3f28609253817be26d3572ec7eb9ce40129ecd6f4f6693ef"
Jul 22 02:30:25.228552 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:25.228521    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:30:25.228650 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:25.228601    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:30:25.228713 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:25.228658    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:30:26.212167 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:26.212139    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-76d9db859c-rhpqn"
Jul 22 02:30:26.240779 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:26.240738    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp" event={"ID":"3b40b006-204b-4291-901e-af512339c609","Type":"ContainerStarted","Data":"3e6d2828c7f6754f37788af1d2b0ebf59522ba65b3b6ed9e5bdbd53beeea70d5"}
Jul 22 02:30:26.518143 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:26.518044    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:30:26.518446 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:26.518423    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-xw2bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Jul 22 02:30:26.518603 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:26.518575    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp" podUID="3b40b006-204b-4291-901e-af512339c609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Jul 22 02:30:27.233314 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:27.233235    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:30:27.517696 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:27.517603    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-xw2bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Jul 22 02:30:27.517696 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:27.517673    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp" podUID="3b40b006-204b-4291-901e-af512339c609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Jul 22 02:30:28.517860 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:28.517803    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-xw2bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Jul 22 02:30:28.518359 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:28.517913    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp" podUID="3b40b006-204b-4291-901e-af512339c609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Jul 22 02:30:28.738405 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:28.738319    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:30:29.518478 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:29.517991    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-xw2bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Jul 22 02:30:29.518478 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:29.518073    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp" podUID="3b40b006-204b-4291-901e-af512339c609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Jul 22 02:30:30.522160 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:30.522127    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:30:30.522541 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:30.522521    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:30:30.523161 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:30.523145    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-xw2bp"
Jul 22 02:30:33.218581 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:33.218543    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-vdmlh"
Jul 22 02:30:33.908742 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:33.908716    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:30:50.207393 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:50.207356    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:30:58.376592 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.376557    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"]
Jul 22 02:30:58.384044 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.384020    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:30:58.387422 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.387397    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dz5z5\""
Jul 22 02:30:58.387422 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.387394    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 22 02:30:58.387629 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.387467    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 22 02:30:58.387629 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.387499    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 22 02:30:58.393087 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.393056    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"]
Jul 22 02:30:58.416858 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.416826    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:30:58.481264 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.481231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e59b6f1-67b6-4bc1-8535-df5ca681b7e1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pvmpd\" (UID: \"0e59b6f1-67b6-4bc1-8535-df5ca681b7e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:30:58.481422 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.481290    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e59b6f1-67b6-4bc1-8535-df5ca681b7e1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pvmpd\" (UID: \"0e59b6f1-67b6-4bc1-8535-df5ca681b7e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:30:58.481422 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.481328    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e59b6f1-67b6-4bc1-8535-df5ca681b7e1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pvmpd\" (UID: \"0e59b6f1-67b6-4bc1-8535-df5ca681b7e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:30:58.481422 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.481354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx75v\" (UniqueName: \"kubernetes.io/projected/0e59b6f1-67b6-4bc1-8535-df5ca681b7e1-kube-api-access-tx75v\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pvmpd\" (UID: \"0e59b6f1-67b6-4bc1-8535-df5ca681b7e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:30:58.481422 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.481404    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e59b6f1-67b6-4bc1-8535-df5ca681b7e1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pvmpd\" (UID: \"0e59b6f1-67b6-4bc1-8535-df5ca681b7e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:30:58.481555 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.481433    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e59b6f1-67b6-4bc1-8535-df5ca681b7e1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pvmpd\" (UID: \"0e59b6f1-67b6-4bc1-8535-df5ca681b7e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:30:58.581870 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.581801    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e59b6f1-67b6-4bc1-8535-df5ca681b7e1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pvmpd\" (UID: \"0e59b6f1-67b6-4bc1-8535-df5ca681b7e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:30:58.582075 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.581892    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e59b6f1-67b6-4bc1-8535-df5ca681b7e1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pvmpd\" (UID: \"0e59b6f1-67b6-4bc1-8535-df5ca681b7e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:30:58.582075 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.581919    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e59b6f1-67b6-4bc1-8535-df5ca681b7e1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pvmpd\" (UID: \"0e59b6f1-67b6-4bc1-8535-df5ca681b7e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:30:58.582075 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.581946    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx75v\" (UniqueName: \"kubernetes.io/projected/0e59b6f1-67b6-4bc1-8535-df5ca681b7e1-kube-api-access-tx75v\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pvmpd\" (UID: \"0e59b6f1-67b6-4bc1-8535-df5ca681b7e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:30:58.582075 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.581974    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e59b6f1-67b6-4bc1-8535-df5ca681b7e1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pvmpd\" (UID: \"0e59b6f1-67b6-4bc1-8535-df5ca681b7e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:30:58.582075 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.581995    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e59b6f1-67b6-4bc1-8535-df5ca681b7e1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pvmpd\" (UID: \"0e59b6f1-67b6-4bc1-8535-df5ca681b7e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:30:58.582339 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.582289    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e59b6f1-67b6-4bc1-8535-df5ca681b7e1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pvmpd\" (UID: \"0e59b6f1-67b6-4bc1-8535-df5ca681b7e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:30:58.582417 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.582396    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e59b6f1-67b6-4bc1-8535-df5ca681b7e1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pvmpd\" (UID: \"0e59b6f1-67b6-4bc1-8535-df5ca681b7e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:30:58.582538 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.582524    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e59b6f1-67b6-4bc1-8535-df5ca681b7e1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pvmpd\" (UID: \"0e59b6f1-67b6-4bc1-8535-df5ca681b7e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:30:58.584582 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.584555    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e59b6f1-67b6-4bc1-8535-df5ca681b7e1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pvmpd\" (UID: \"0e59b6f1-67b6-4bc1-8535-df5ca681b7e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:30:58.584698 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.584683    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e59b6f1-67b6-4bc1-8535-df5ca681b7e1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pvmpd\" (UID: \"0e59b6f1-67b6-4bc1-8535-df5ca681b7e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:30:58.591348 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.591303    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx75v\" (UniqueName: \"kubernetes.io/projected/0e59b6f1-67b6-4bc1-8535-df5ca681b7e1-kube-api-access-tx75v\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pvmpd\" (UID: \"0e59b6f1-67b6-4bc1-8535-df5ca681b7e1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:30:58.695541 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.695451    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:30:58.833922 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:58.833897    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"]
Jul 22 02:30:58.835770 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:30:58.835743    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e59b6f1_67b6_4bc1_8535_df5ca681b7e1.slice/crio-39fc3ddb1edf8232552bb1a95ed0366302683285874e79d7d3c47c46566e2373 WatchSource:0}: Error finding container 39fc3ddb1edf8232552bb1a95ed0366302683285874e79d7d3c47c46566e2373: Status 404 returned error can't find the container with id 39fc3ddb1edf8232552bb1a95ed0366302683285874e79d7d3c47c46566e2373
Jul 22 02:30:59.390833 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:30:59.390797    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd" event={"ID":"0e59b6f1-67b6-4bc1-8535-df5ca681b7e1","Type":"ContainerStarted","Data":"39fc3ddb1edf8232552bb1a95ed0366302683285874e79d7d3c47c46566e2373"}
Jul 22 02:31:00.405665 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:00.405625    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd" event={"ID":"0e59b6f1-67b6-4bc1-8535-df5ca681b7e1","Type":"ContainerStarted","Data":"9e3dec4fa6847df75bd9dece014de78c9e8c23f35f9487efc3c36683b738b1e9"}
Jul 22 02:31:00.431554 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:00.431479    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd" podStartSLOduration=1.399417572 podStartE2EDuration="2.431456523s" totalImagesPullingTime="1.032038951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:31:00.43076341 +0000 UTC m=+611.279099985" watchObservedRunningTime="2026-07-22 02:31:00.431456523 +0000 UTC m=+611.279793099"
Jul 22 02:31:08.695907 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:08.695870    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:31:08.696289 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:08.695934    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:31:08.708605 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:08.708584    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:31:09.458752 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:09.458723    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pvmpd"
Jul 22 02:31:15.403577 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:15.403482    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:31:15.485519 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:15.485482    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:31:17.076309 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:17.076273    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:31:36.533339 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.533304    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"]
Jul 22 02:31:36.537473 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.537443    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:36.541989 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.541965    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 22 02:31:36.556303 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.556278    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"]
Jul 22 02:31:36.612986 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.612954    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:31:36.708024 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.707990    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcb6p\" (UniqueName: \"kubernetes.io/projected/c9bfbdeb-f030-48d0-a8a4-4f660c937c48-kube-api-access-pcb6p\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k9zkf\" (UID: \"c9bfbdeb-f030-48d0-a8a4-4f660c937c48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:36.708024 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.708027    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9bfbdeb-f030-48d0-a8a4-4f660c937c48-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k9zkf\" (UID: \"c9bfbdeb-f030-48d0-a8a4-4f660c937c48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:36.708232 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.708129    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9bfbdeb-f030-48d0-a8a4-4f660c937c48-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k9zkf\" (UID: \"c9bfbdeb-f030-48d0-a8a4-4f660c937c48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:36.708232 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.708175    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9bfbdeb-f030-48d0-a8a4-4f660c937c48-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k9zkf\" (UID: \"c9bfbdeb-f030-48d0-a8a4-4f660c937c48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:36.708232 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.708197    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9bfbdeb-f030-48d0-a8a4-4f660c937c48-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k9zkf\" (UID: \"c9bfbdeb-f030-48d0-a8a4-4f660c937c48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:36.708340 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.708242    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9bfbdeb-f030-48d0-a8a4-4f660c937c48-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k9zkf\" (UID: \"c9bfbdeb-f030-48d0-a8a4-4f660c937c48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:36.809489 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.809408    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9bfbdeb-f030-48d0-a8a4-4f660c937c48-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k9zkf\" (UID: \"c9bfbdeb-f030-48d0-a8a4-4f660c937c48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:36.809489 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.809459    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcb6p\" (UniqueName: \"kubernetes.io/projected/c9bfbdeb-f030-48d0-a8a4-4f660c937c48-kube-api-access-pcb6p\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k9zkf\" (UID: \"c9bfbdeb-f030-48d0-a8a4-4f660c937c48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:36.809489 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.809477    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9bfbdeb-f030-48d0-a8a4-4f660c937c48-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k9zkf\" (UID: \"c9bfbdeb-f030-48d0-a8a4-4f660c937c48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:36.809725 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.809531    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9bfbdeb-f030-48d0-a8a4-4f660c937c48-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k9zkf\" (UID: \"c9bfbdeb-f030-48d0-a8a4-4f660c937c48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:36.809725 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.809556    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9bfbdeb-f030-48d0-a8a4-4f660c937c48-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k9zkf\" (UID: \"c9bfbdeb-f030-48d0-a8a4-4f660c937c48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:36.809725 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.809579    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9bfbdeb-f030-48d0-a8a4-4f660c937c48-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k9zkf\" (UID: \"c9bfbdeb-f030-48d0-a8a4-4f660c937c48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:36.809940 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.809922    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9bfbdeb-f030-48d0-a8a4-4f660c937c48-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k9zkf\" (UID: \"c9bfbdeb-f030-48d0-a8a4-4f660c937c48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:36.810016 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.809967    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9bfbdeb-f030-48d0-a8a4-4f660c937c48-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k9zkf\" (UID: \"c9bfbdeb-f030-48d0-a8a4-4f660c937c48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:36.810075 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.810022    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9bfbdeb-f030-48d0-a8a4-4f660c937c48-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k9zkf\" (UID: \"c9bfbdeb-f030-48d0-a8a4-4f660c937c48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:36.811828 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.811808    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9bfbdeb-f030-48d0-a8a4-4f660c937c48-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k9zkf\" (UID: \"c9bfbdeb-f030-48d0-a8a4-4f660c937c48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:36.812089 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.812070    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9bfbdeb-f030-48d0-a8a4-4f660c937c48-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k9zkf\" (UID: \"c9bfbdeb-f030-48d0-a8a4-4f660c937c48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:36.819650 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.819627    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcb6p\" (UniqueName: \"kubernetes.io/projected/c9bfbdeb-f030-48d0-a8a4-4f660c937c48-kube-api-access-pcb6p\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k9zkf\" (UID: \"c9bfbdeb-f030-48d0-a8a4-4f660c937c48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:36.850212 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.850185    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:36.991139 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.991112    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"]
Jul 22 02:31:36.992884 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:31:36.992844    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9bfbdeb_f030_48d0_a8a4_4f660c937c48.slice/crio-e8d793602f276da039dbf8d9b91fdd129739cbae9847815a82c977d3e51c3651 WatchSource:0}: Error finding container e8d793602f276da039dbf8d9b91fdd129739cbae9847815a82c977d3e51c3651: Status 404 returned error can't find the container with id e8d793602f276da039dbf8d9b91fdd129739cbae9847815a82c977d3e51c3651
Jul 22 02:31:36.994168 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:36.994151    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 02:31:37.573961 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:37.573868    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf" event={"ID":"c9bfbdeb-f030-48d0-a8a4-4f660c937c48","Type":"ContainerStarted","Data":"661ef4615b8035c20a276099cb7472522058f4355696d4e624f803f1298d6d3c"}
Jul 22 02:31:37.573961 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:37.573931    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf" event={"ID":"c9bfbdeb-f030-48d0-a8a4-4f660c937c48","Type":"ContainerStarted","Data":"e8d793602f276da039dbf8d9b91fdd129739cbae9847815a82c977d3e51c3651"}
Jul 22 02:31:37.607059 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:37.607007    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf" podStartSLOduration=1.452036301 podStartE2EDuration="1.606990147s" totalImagesPullingTime="154.953846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:31:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:31:37.605167398 +0000 UTC m=+648.453503972" watchObservedRunningTime="2026-07-22 02:31:37.606990147 +0000 UTC m=+648.455326721"
Jul 22 02:31:46.851092 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:46.851054    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:46.851458 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:46.851124    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:46.863077 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:46.863055    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:31:47.629275 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:31:47.629240    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k9zkf"
Jul 22 02:32:04.992690 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:32:04.992656    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:32:07.041976 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:32:07.041943    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:32:39.366814 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:32:39.366782    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:32:55.709555 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:32:55.709477    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:33:03.276570 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:03.276522    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-866d947867-vftd7"]
Jul 22 02:33:03.277102 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:03.276781    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-866d947867-vftd7" podUID="d141a5c2-91b4-4fff-833d-ef677140bcd1" containerName="authorino" containerID="cri-o://55e22bb7334ef19c2a1f00b4f341da23ed14f71917a56ab26cdd0d2d6a6ee213" gracePeriod=30
Jul 22 02:33:03.521085 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:03.521063    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-866d947867-vftd7"
Jul 22 02:33:03.613019 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:03.612992    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8lvw\" (UniqueName: \"kubernetes.io/projected/d141a5c2-91b4-4fff-833d-ef677140bcd1-kube-api-access-n8lvw\") pod \"d141a5c2-91b4-4fff-833d-ef677140bcd1\" (UID: \"d141a5c2-91b4-4fff-833d-ef677140bcd1\") "
Jul 22 02:33:03.613169 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:03.613044    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d141a5c2-91b4-4fff-833d-ef677140bcd1-tls-cert\") pod \"d141a5c2-91b4-4fff-833d-ef677140bcd1\" (UID: \"d141a5c2-91b4-4fff-833d-ef677140bcd1\") "
Jul 22 02:33:03.615068 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:03.615039    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d141a5c2-91b4-4fff-833d-ef677140bcd1-kube-api-access-n8lvw" (OuterVolumeSpecName: "kube-api-access-n8lvw") pod "d141a5c2-91b4-4fff-833d-ef677140bcd1" (UID: "d141a5c2-91b4-4fff-833d-ef677140bcd1"). InnerVolumeSpecName "kube-api-access-n8lvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:33:03.624360 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:03.624335    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d141a5c2-91b4-4fff-833d-ef677140bcd1-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d141a5c2-91b4-4fff-833d-ef677140bcd1" (UID: "d141a5c2-91b4-4fff-833d-ef677140bcd1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:33:03.714552 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:03.714521    2565 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d141a5c2-91b4-4fff-833d-ef677140bcd1-tls-cert\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:33:03.714552 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:03.714548    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8lvw\" (UniqueName: \"kubernetes.io/projected/d141a5c2-91b4-4fff-833d-ef677140bcd1-kube-api-access-n8lvw\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:33:03.819872 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:03.819813    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:33:03.915334 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:03.915259    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:33:03.950977 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:03.950945    2565 generic.go:356] "Generic (PLEG): container finished" podID="d141a5c2-91b4-4fff-833d-ef677140bcd1" containerID="55e22bb7334ef19c2a1f00b4f341da23ed14f71917a56ab26cdd0d2d6a6ee213" exitCode=0
Jul 22 02:33:03.951124 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:03.950989    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-866d947867-vftd7" event={"ID":"d141a5c2-91b4-4fff-833d-ef677140bcd1","Type":"ContainerDied","Data":"55e22bb7334ef19c2a1f00b4f341da23ed14f71917a56ab26cdd0d2d6a6ee213"}
Jul 22 02:33:03.951124 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:03.950994    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-866d947867-vftd7"
Jul 22 02:33:03.951124 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:03.951028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-866d947867-vftd7" event={"ID":"d141a5c2-91b4-4fff-833d-ef677140bcd1","Type":"ContainerDied","Data":"63786980a60a42f9516d652bfc70e1f56fa814512d125a247880b6528c1f721f"}
Jul 22 02:33:03.951124 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:03.951054    2565 scope.go:117] "RemoveContainer" containerID="55e22bb7334ef19c2a1f00b4f341da23ed14f71917a56ab26cdd0d2d6a6ee213"
Jul 22 02:33:03.959362 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:03.959342    2565 scope.go:117] "RemoveContainer" containerID="55e22bb7334ef19c2a1f00b4f341da23ed14f71917a56ab26cdd0d2d6a6ee213"
Jul 22 02:33:03.959641 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:33:03.959620    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55e22bb7334ef19c2a1f00b4f341da23ed14f71917a56ab26cdd0d2d6a6ee213\": container with ID starting with 55e22bb7334ef19c2a1f00b4f341da23ed14f71917a56ab26cdd0d2d6a6ee213 not found: ID does not exist" containerID="55e22bb7334ef19c2a1f00b4f341da23ed14f71917a56ab26cdd0d2d6a6ee213"
Jul 22 02:33:03.959714 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:03.959650    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55e22bb7334ef19c2a1f00b4f341da23ed14f71917a56ab26cdd0d2d6a6ee213"} err="failed to get container status \"55e22bb7334ef19c2a1f00b4f341da23ed14f71917a56ab26cdd0d2d6a6ee213\": rpc error: code = NotFound desc = could not find container \"55e22bb7334ef19c2a1f00b4f341da23ed14f71917a56ab26cdd0d2d6a6ee213\": container with ID starting with 55e22bb7334ef19c2a1f00b4f341da23ed14f71917a56ab26cdd0d2d6a6ee213 not found: ID does not exist"
Jul 22 02:33:03.968582 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:03.968560    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-866d947867-vftd7"]
Jul 22 02:33:03.972321 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:03.972301    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-866d947867-vftd7"]
Jul 22 02:33:04.964212 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:04.964169    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:33:05.059959 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:05.059927    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:33:05.726150 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:33:05.726113    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d141a5c2-91b4-4fff-833d-ef677140bcd1" path="/var/lib/kubelet/pods/d141a5c2-91b4-4fff-833d-ef677140bcd1/volumes"
Jul 22 02:34:03.047198 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:34:03.047167    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:34:03.147451 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:34:03.147423    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:34:17.409956 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:34:17.409869    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:35:09.474301 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:09.474268    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:35:10.700049 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:10.700015    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:35:10.826561 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:10.826526    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b897bdc49-pm7x6"]
Jul 22 02:35:10.826909 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:10.826897    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d141a5c2-91b4-4fff-833d-ef677140bcd1" containerName="authorino"
Jul 22 02:35:10.826962 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:10.826910    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d141a5c2-91b4-4fff-833d-ef677140bcd1" containerName="authorino"
Jul 22 02:35:10.826998 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:10.826980    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d141a5c2-91b4-4fff-833d-ef677140bcd1" containerName="authorino"
Jul 22 02:35:10.828965 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:10.828946    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b897bdc49-pm7x6"
Jul 22 02:35:10.832341 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:10.832325    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 22 02:35:10.832341 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:10.832336    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rddtn\""
Jul 22 02:35:10.832494 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:10.832337    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 22 02:35:10.840282 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:10.840261    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b897bdc49-pm7x6"]
Jul 22 02:35:10.866176 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:10.866149    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fafd6279-1ccd-4c2e-b5ee-88876154879b-webhook-cert\") pod \"maas-controller-5b897bdc49-pm7x6\" (UID: \"fafd6279-1ccd-4c2e-b5ee-88876154879b\") " pod="opendatahub/maas-controller-5b897bdc49-pm7x6"
Jul 22 02:35:10.866274 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:10.866185    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnns8\" (UniqueName: \"kubernetes.io/projected/fafd6279-1ccd-4c2e-b5ee-88876154879b-kube-api-access-dnns8\") pod \"maas-controller-5b897bdc49-pm7x6\" (UID: \"fafd6279-1ccd-4c2e-b5ee-88876154879b\") " pod="opendatahub/maas-controller-5b897bdc49-pm7x6"
Jul 22 02:35:10.966730 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:10.966658    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fafd6279-1ccd-4c2e-b5ee-88876154879b-webhook-cert\") pod \"maas-controller-5b897bdc49-pm7x6\" (UID: \"fafd6279-1ccd-4c2e-b5ee-88876154879b\") " pod="opendatahub/maas-controller-5b897bdc49-pm7x6"
Jul 22 02:35:10.966730 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:10.966704    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnns8\" (UniqueName: \"kubernetes.io/projected/fafd6279-1ccd-4c2e-b5ee-88876154879b-kube-api-access-dnns8\") pod \"maas-controller-5b897bdc49-pm7x6\" (UID: \"fafd6279-1ccd-4c2e-b5ee-88876154879b\") " pod="opendatahub/maas-controller-5b897bdc49-pm7x6"
Jul 22 02:35:10.969120 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:10.969101    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fafd6279-1ccd-4c2e-b5ee-88876154879b-webhook-cert\") pod \"maas-controller-5b897bdc49-pm7x6\" (UID: \"fafd6279-1ccd-4c2e-b5ee-88876154879b\") " pod="opendatahub/maas-controller-5b897bdc49-pm7x6"
Jul 22 02:35:10.975190 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:10.975163    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnns8\" (UniqueName: \"kubernetes.io/projected/fafd6279-1ccd-4c2e-b5ee-88876154879b-kube-api-access-dnns8\") pod \"maas-controller-5b897bdc49-pm7x6\" (UID: \"fafd6279-1ccd-4c2e-b5ee-88876154879b\") " pod="opendatahub/maas-controller-5b897bdc49-pm7x6"
Jul 22 02:35:11.138920 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:11.138886    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b897bdc49-pm7x6"
Jul 22 02:35:11.272101 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:11.272079    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b897bdc49-pm7x6"]
Jul 22 02:35:11.273069 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:35:11.273043    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfafd6279_1ccd_4c2e_b5ee_88876154879b.slice/crio-ee7a1209c61efbcf3c41c4bc3b9134c7831f7f9e35c46175cc7b8784db53d5df WatchSource:0}: Error finding container ee7a1209c61efbcf3c41c4bc3b9134c7831f7f9e35c46175cc7b8784db53d5df: Status 404 returned error can't find the container with id ee7a1209c61efbcf3c41c4bc3b9134c7831f7f9e35c46175cc7b8784db53d5df
Jul 22 02:35:11.506950 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:11.506843    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b897bdc49-pm7x6" event={"ID":"fafd6279-1ccd-4c2e-b5ee-88876154879b","Type":"ContainerStarted","Data":"ee7a1209c61efbcf3c41c4bc3b9134c7831f7f9e35c46175cc7b8784db53d5df"}
Jul 22 02:35:12.513729 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:12.513692    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b897bdc49-pm7x6" event={"ID":"fafd6279-1ccd-4c2e-b5ee-88876154879b","Type":"ContainerStarted","Data":"f056508ab872af605eb53aebfe510a4346a9c7bccbd356635f9f096edbe7686d"}
Jul 22 02:35:12.514131 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:12.513792    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b897bdc49-pm7x6"
Jul 22 02:35:12.531247 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:12.531204    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b897bdc49-pm7x6" podStartSLOduration=2.205649953 podStartE2EDuration="2.531188882s" totalImagesPullingTime="325.538929ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:35:12.529466052 +0000 UTC m=+863.377802626" watchObservedRunningTime="2026-07-22 02:35:12.531188882 +0000 UTC m=+863.379525456"
Jul 22 02:35:12.888649 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:12.888617    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:35:12.989182 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:12.989153    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:35:23.521535 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:23.521505    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b897bdc49-pm7x6"
Jul 22 02:35:51.895197 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:51.895167    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:35:53.431972 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:53.431939    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:35:58.414904 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:58.414873    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:35:58.491553 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:58.491528    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:35:58.596044 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:58.596012    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:35:58.687609 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:58.687530    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:35:58.794101 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:35:58.794061    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:36:04.689359 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:36:04.689327    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:36:04.758123 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:36:04.758076    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:39:40.401557 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:39:40.401513    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:39:50.273611 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:39:50.273572    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:40:44.993343 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:40:44.993308    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:40:45.105632 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:40:45.105596    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:42:01.310626 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:42:01.310527    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:42:02.860045 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:42:02.860007    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:43:44.593087 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.592998    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"]
Jul 22 02:43:44.596513 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.596497    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:44.599776 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.599756    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-bec2a4d0-kserve-self-signed-certs\""
Jul 22 02:43:44.629393 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.629363    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"]
Jul 22 02:43:44.634592 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.634568    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-home\") pod \"e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") " pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:44.634718 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.634600    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-tls-certs\") pod \"e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") " pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:44.634718 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.634640    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-dshm\") pod \"e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") " pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:44.634718 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.634688    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-model-cache\") pod \"e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") " pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:44.634718 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.634714    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx574\" (UniqueName: \"kubernetes.io/projected/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-kube-api-access-mx574\") pod \"e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") " pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:44.634886 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.634737    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-tmp-dir\") pod \"e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") " pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:44.687766 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.687728    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:43:44.736027 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.735990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-home\") pod \"e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") " pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:44.736027 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.736032    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-tls-certs\") pod \"e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") " pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:44.736263 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.736073    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-dshm\") pod \"e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") " pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:44.736263 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.736129    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-model-cache\") pod \"e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") " pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:44.736263 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.736173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx574\" (UniqueName: \"kubernetes.io/projected/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-kube-api-access-mx574\") pod \"e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") " pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:44.736263 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.736208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-tmp-dir\") pod \"e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") " pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:44.736475 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.736455    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-home\") pod \"e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") " pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:44.736583 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.736563    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-model-cache\") pod \"e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") " pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:44.736646 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.736600    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-tmp-dir\") pod \"e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") " pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:44.738438 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.738423    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-dshm\") pod \"e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") " pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:44.738560 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.738544    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-tls-certs\") pod \"e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") " pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:44.745164 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.745141    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx574\" (UniqueName: \"kubernetes.io/projected/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-kube-api-access-mx574\") pod \"e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") " pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:44.905741 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:44.905705    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:45.047797 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:45.047765    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"]
Jul 22 02:43:45.052110 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:43:45.049798    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f7a7ecc_5319_4fea_b2e5_2ed0da496ca4.slice/crio-9af5fac2249e5190317755e4c6b84d172040fd5d99f576a265277b95e667f0ce WatchSource:0}: Error finding container 9af5fac2249e5190317755e4c6b84d172040fd5d99f576a265277b95e667f0ce: Status 404 returned error can't find the container with id 9af5fac2249e5190317755e4c6b84d172040fd5d99f576a265277b95e667f0ce
Jul 22 02:43:45.791145 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:45.791114    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb" event={"ID":"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4","Type":"ContainerStarted","Data":"864b19489f277487db4cdef2b0ace406fb5c1e5e039339174f42c8672f29457f"}
Jul 22 02:43:45.791502 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:45.791154    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb" event={"ID":"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4","Type":"ContainerStarted","Data":"9af5fac2249e5190317755e4c6b84d172040fd5d99f576a265277b95e667f0ce"}
Jul 22 02:43:45.810927 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:45.810877    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb" podStartSLOduration=1.810862405 podStartE2EDuration="1.810862405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:43:45.808974935 +0000 UTC m=+1376.657311508" watchObservedRunningTime="2026-07-22 02:43:45.810862405 +0000 UTC m=+1376.659198969"
Jul 22 02:43:46.985951 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:46.985905    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:43:54.906287 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:54.906257    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:54.906646 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:54.906306    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:54.918724 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:54.918701    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:43:55.844084 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:43:55.844055    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:44:37.576359 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:37.576328    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"]
Jul 22 02:44:37.608281 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:37.608247    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:44:39.101052 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:39.101020    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:44:52.597456 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:52.597375    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb" podUID="9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4" containerName="main" containerID="cri-o://864b19489f277487db4cdef2b0ace406fb5c1e5e039339174f42c8672f29457f" gracePeriod=45
Jul 22 02:44:52.857087 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:52.857063    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:44:52.898036 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:52.897808    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-home\") pod \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") "
Jul 22 02:44:52.898036 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:52.897877    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-dshm\") pod \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") "
Jul 22 02:44:52.898036 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:52.897928    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-tls-certs\") pod \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") "
Jul 22 02:44:52.898036 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:52.897982    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx574\" (UniqueName: \"kubernetes.io/projected/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-kube-api-access-mx574\") pod \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") "
Jul 22 02:44:52.898339 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:52.898049    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-model-cache\") pod \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") "
Jul 22 02:44:52.898339 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:52.898085    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-tmp-dir\") pod \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\" (UID: \"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4\") "
Jul 22 02:44:52.898579 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:52.898552    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4" (UID: "9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:44:52.898710 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:52.898651    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-home" (OuterVolumeSpecName: "home") pod "9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4" (UID: "9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:44:52.898778 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:52.898708    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4" (UID: "9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:44:52.900626 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:52.900573    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-kube-api-access-mx574" (OuterVolumeSpecName: "kube-api-access-mx574") pod "9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4" (UID: "9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4"). InnerVolumeSpecName "kube-api-access-mx574". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:44:52.900626 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:52.900608    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-dshm" (OuterVolumeSpecName: "dshm") pod "9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4" (UID: "9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:44:52.900788 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:52.900662    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4" (UID: "9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:44:52.998988 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:52.998950    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx574\" (UniqueName: \"kubernetes.io/projected/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-kube-api-access-mx574\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:44:52.998988 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:52.998984    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-model-cache\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:44:52.999160 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:52.999015    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-tmp-dir\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:44:52.999160 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:52.999023    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-home\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:44:52.999160 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:52.999031    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-dshm\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:44:52.999160 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:52.999038    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4-tls-certs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:44:53.094020 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:53.093987    2565 generic.go:356] "Generic (PLEG): container finished" podID="9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4" containerID="864b19489f277487db4cdef2b0ace406fb5c1e5e039339174f42c8672f29457f" exitCode=0
Jul 22 02:44:53.094163 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:53.094028    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"
Jul 22 02:44:53.094163 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:53.094064    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb" event={"ID":"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4","Type":"ContainerDied","Data":"864b19489f277487db4cdef2b0ace406fb5c1e5e039339174f42c8672f29457f"}
Jul 22 02:44:53.094163 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:53.094113    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb" event={"ID":"9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4","Type":"ContainerDied","Data":"9af5fac2249e5190317755e4c6b84d172040fd5d99f576a265277b95e667f0ce"}
Jul 22 02:44:53.094163 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:53.094132    2565 scope.go:117] "RemoveContainer" containerID="864b19489f277487db4cdef2b0ace406fb5c1e5e039339174f42c8672f29457f"
Jul 22 02:44:53.103058 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:53.103043    2565 scope.go:117] "RemoveContainer" containerID="864b19489f277487db4cdef2b0ace406fb5c1e5e039339174f42c8672f29457f"
Jul 22 02:44:53.103308 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:44:53.103290    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"864b19489f277487db4cdef2b0ace406fb5c1e5e039339174f42c8672f29457f\": container with ID starting with 864b19489f277487db4cdef2b0ace406fb5c1e5e039339174f42c8672f29457f not found: ID does not exist" containerID="864b19489f277487db4cdef2b0ace406fb5c1e5e039339174f42c8672f29457f"
Jul 22 02:44:53.103373 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:53.103316    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"864b19489f277487db4cdef2b0ace406fb5c1e5e039339174f42c8672f29457f"} err="failed to get container status \"864b19489f277487db4cdef2b0ace406fb5c1e5e039339174f42c8672f29457f\": rpc error: code = NotFound desc = could not find container \"864b19489f277487db4cdef2b0ace406fb5c1e5e039339174f42c8672f29457f\": container with ID starting with 864b19489f277487db4cdef2b0ace406fb5c1e5e039339174f42c8672f29457f not found: ID does not exist"
Jul 22 02:44:53.117512 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:53.117467    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"]
Jul 22 02:44:53.119181 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:53.119163    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-bec2a4d0-kserve-569f9b6475-zcxjb"]
Jul 22 02:44:53.726294 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:44:53.726264    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4" path="/var/lib/kubelet/pods/9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4/volumes"
Jul 22 02:47:46.221096 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:47:46.221018    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:48:06.801545 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:06.801510    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:48:09.861299 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:09.861269    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:48:09.957879 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:09.957832    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:48:10.344590 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:10.344511    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:48:16.183881 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.183832    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:48:16.262831 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.262798    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:48:16.436051 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.435960    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:48:16.720839 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.720752    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"]
Jul 22 02:48:16.721198 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.721181    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4" containerName="main"
Jul 22 02:48:16.721245 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.721200    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4" containerName="main"
Jul 22 02:48:16.721284 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.721263    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f7a7ecc-5319-4fea-b2e5-2ed0da496ca4" containerName="main"
Jul 22 02:48:16.723807 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.723788    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.726143 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.726124    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e3ec0889-openshift-default-dockercfg-zdp9m\""
Jul 22 02:48:16.747586 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.747565    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"]
Jul 22 02:48:16.751422 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.751400    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:48:16.807352 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.807317    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-workload-socket\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.807565 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.807359    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-istio-envoy\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.807565 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.807405    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-credential-socket\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.807565 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.807481    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d213f17-a47f-406e-b444-b96a99215261-istio-token\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.807565 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.807550    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-workload-certs\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.807714 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.807580    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-istio-data\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.807714 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.807658    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d213f17-a47f-406e-b444-b96a99215261-istiod-ca-cert\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.807779 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.807734    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d213f17-a47f-406e-b444-b96a99215261-istio-podinfo\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.807838 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.807821    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fjct\" (UniqueName: \"kubernetes.io/projected/3d213f17-a47f-406e-b444-b96a99215261-kube-api-access-6fjct\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.909038 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.909003    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-workload-certs\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.909038 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.909047    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-istio-data\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.909291 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.909088    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d213f17-a47f-406e-b444-b96a99215261-istiod-ca-cert\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.909291 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.909119    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d213f17-a47f-406e-b444-b96a99215261-istio-podinfo\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.909291 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.909177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fjct\" (UniqueName: \"kubernetes.io/projected/3d213f17-a47f-406e-b444-b96a99215261-kube-api-access-6fjct\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.909291 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.909209    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-workload-socket\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.909291 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.909237    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-istio-envoy\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.909291 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.909281    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-credential-socket\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.909593 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.909311    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d213f17-a47f-406e-b444-b96a99215261-istio-token\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.909593 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.909457    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-workload-certs\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.909593 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.909566    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-workload-socket\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.909780 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.909750    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-credential-socket\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.910040 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.910016    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d213f17-a47f-406e-b444-b96a99215261-istiod-ca-cert\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.910040 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.910035    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-istio-data\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.911690 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.911669    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-istio-envoy\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.912084 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.912062    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d213f17-a47f-406e-b444-b96a99215261-istio-podinfo\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.919863 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.919827    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d213f17-a47f-406e-b444-b96a99215261-istio-token\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:16.919952 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:16.919882    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fjct\" (UniqueName: \"kubernetes.io/projected/3d213f17-a47f-406e-b444-b96a99215261-kube-api-access-6fjct\") pod \"e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") " pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:17.026411 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:17.026334    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:48:17.034346 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:17.034327    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:17.172916 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:17.172800    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"]
Jul 22 02:48:17.177099 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:48:17.177065    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d213f17_a47f_406e_b444_b96a99215261.slice/crio-c2844909ba373640994d89468dd1c1293b6217e0dc744bbcc3b452d7ef7db341 WatchSource:0}: Error finding container c2844909ba373640994d89468dd1c1293b6217e0dc744bbcc3b452d7ef7db341: Status 404 returned error can't find the container with id c2844909ba373640994d89468dd1c1293b6217e0dc744bbcc3b452d7ef7db341
Jul 22 02:48:17.179200 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:17.179167    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:48:17.179295 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:17.179231    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:48:17.179295 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:17.179260    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:48:18.010002 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:18.009968    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w" event={"ID":"3d213f17-a47f-406e-b444-b96a99215261","Type":"ContainerStarted","Data":"3163d23bbc8a7f3df0f2d26c81adc358aa3f6a0a33cdb45bb362f567150ce2e5"}
Jul 22 02:48:18.010462 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:18.010020    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w" event={"ID":"3d213f17-a47f-406e-b444-b96a99215261","Type":"ContainerStarted","Data":"c2844909ba373640994d89468dd1c1293b6217e0dc744bbcc3b452d7ef7db341"}
Jul 22 02:48:18.035301 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:18.035273    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:18.039933 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:18.039911    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:18.045205 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:18.045164    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w" podStartSLOduration=2.045151319 podStartE2EDuration="2.045151319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:48:18.043160918 +0000 UTC m=+1648.891497492" watchObservedRunningTime="2026-07-22 02:48:18.045151319 +0000 UTC m=+1648.893487892"
Jul 22 02:48:19.014927 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:19.014891    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:19.015860 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:19.015823    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:19.449901 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:19.449844    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:48:19.521272 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:19.521212    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"]
Jul 22 02:48:19.525393 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:19.525371    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"
Jul 22 02:48:19.528703 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:19.528681    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e3ec0889-serving-cert\""
Jul 22 02:48:19.544521 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:19.544498    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"]
Jul 22 02:48:19.580060 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:19.580036    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:48:19.636218 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:19.636181    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59b70d3-4b1f-448c-83de-9010afd3ab7c-openshift-service-ca\") pod \"maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq\" (UID: \"c59b70d3-4b1f-448c-83de-9010afd3ab7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"
Jul 22 02:48:19.636480 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:19.636306    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c59b70d3-4b1f-448c-83de-9010afd3ab7c-maas-api-tls\") pod \"maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq\" (UID: \"c59b70d3-4b1f-448c-83de-9010afd3ab7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"
Jul 22 02:48:19.636480 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:19.636374    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txbzq\" (UniqueName: \"kubernetes.io/projected/c59b70d3-4b1f-448c-83de-9010afd3ab7c-kube-api-access-txbzq\") pod \"maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq\" (UID: \"c59b70d3-4b1f-448c-83de-9010afd3ab7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"
Jul 22 02:48:19.680496 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:19.680443    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:48:19.737536 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:19.737461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59b70d3-4b1f-448c-83de-9010afd3ab7c-openshift-service-ca\") pod \"maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq\" (UID: \"c59b70d3-4b1f-448c-83de-9010afd3ab7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"
Jul 22 02:48:19.737536 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:19.737524    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c59b70d3-4b1f-448c-83de-9010afd3ab7c-maas-api-tls\") pod \"maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq\" (UID: \"c59b70d3-4b1f-448c-83de-9010afd3ab7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"
Jul 22 02:48:19.737723 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:19.737572    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txbzq\" (UniqueName: \"kubernetes.io/projected/c59b70d3-4b1f-448c-83de-9010afd3ab7c-kube-api-access-txbzq\") pod \"maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq\" (UID: \"c59b70d3-4b1f-448c-83de-9010afd3ab7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"
Jul 22 02:48:19.737723 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:48:19.737692    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-serving-cert: secret "maas-api-e2e-ait-e3ec0889-serving-cert" not found
Jul 22 02:48:19.737825 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:48:19.737758    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59b70d3-4b1f-448c-83de-9010afd3ab7c-maas-api-tls podName:c59b70d3-4b1f-448c-83de-9010afd3ab7c nodeName:}" failed. No retries permitted until 2026-07-22 02:48:20.237740376 +0000 UTC m=+1651.086076932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c59b70d3-4b1f-448c-83de-9010afd3ab7c-maas-api-tls") pod "maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq" (UID: "c59b70d3-4b1f-448c-83de-9010afd3ab7c") : secret "maas-api-e2e-ait-e3ec0889-serving-cert" not found
Jul 22 02:48:19.738113 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:19.738096    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59b70d3-4b1f-448c-83de-9010afd3ab7c-openshift-service-ca\") pod \"maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq\" (UID: \"c59b70d3-4b1f-448c-83de-9010afd3ab7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"
Jul 22 02:48:19.756875 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:19.756822    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txbzq\" (UniqueName: \"kubernetes.io/projected/c59b70d3-4b1f-448c-83de-9010afd3ab7c-kube-api-access-txbzq\") pod \"maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq\" (UID: \"c59b70d3-4b1f-448c-83de-9010afd3ab7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"
Jul 22 02:48:20.001389 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:20.001314    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:48:20.192376 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:20.192344    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:48:20.242417 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:20.242382    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c59b70d3-4b1f-448c-83de-9010afd3ab7c-maas-api-tls\") pod \"maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq\" (UID: \"c59b70d3-4b1f-448c-83de-9010afd3ab7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"
Jul 22 02:48:20.244756 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:20.244734    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c59b70d3-4b1f-448c-83de-9010afd3ab7c-maas-api-tls\") pod \"maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq\" (UID: \"c59b70d3-4b1f-448c-83de-9010afd3ab7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"
Jul 22 02:48:20.294435 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:20.294360    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:48:20.436103 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:20.436072    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"
Jul 22 02:48:20.575137 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:20.575027    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"]
Jul 22 02:48:20.578409 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:48:20.578379    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59b70d3_4b1f_448c_83de_9010afd3ab7c.slice/crio-ce0d3812e12c81756b8c70150a6d6509ca4bcba4ecc9f73faf3048fea810ad06 WatchSource:0}: Error finding container ce0d3812e12c81756b8c70150a6d6509ca4bcba4ecc9f73faf3048fea810ad06: Status 404 returned error can't find the container with id ce0d3812e12c81756b8c70150a6d6509ca4bcba4ecc9f73faf3048fea810ad06
Jul 22 02:48:20.580269 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:20.580246    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 02:48:21.026307 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:21.026276    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq" event={"ID":"c59b70d3-4b1f-448c-83de-9010afd3ab7c","Type":"ContainerStarted","Data":"56c2bcffbf5d027ed56c4b81b76ad1548f7ba1db27a118d39a9e9d433c105aeb"}
Jul 22 02:48:21.026444 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:21.026332    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq" event={"ID":"c59b70d3-4b1f-448c-83de-9010afd3ab7c","Type":"ContainerStarted","Data":"ce0d3812e12c81756b8c70150a6d6509ca4bcba4ecc9f73faf3048fea810ad06"}
Jul 22 02:48:21.026572 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:21.026548    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"
Jul 22 02:48:21.047657 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:21.047614    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq" podStartSLOduration=1.6668572419999999 podStartE2EDuration="2.047598894s" totalImagesPullingTime="380.741652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:48:21.046500102 +0000 UTC m=+1651.894836677" watchObservedRunningTime="2026-07-22 02:48:21.047598894 +0000 UTC m=+1651.895935468"
Jul 22 02:48:27.042252 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:27.042170    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"
Jul 22 02:48:33.081274 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:33.081186    2565 generic.go:356] "Generic (PLEG): container finished" podID="3d213f17-a47f-406e-b444-b96a99215261" containerID="3163d23bbc8a7f3df0f2d26c81adc358aa3f6a0a33cdb45bb362f567150ce2e5" exitCode=0
Jul 22 02:48:33.081762 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:33.081265    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w" event={"ID":"3d213f17-a47f-406e-b444-b96a99215261","Type":"ContainerDied","Data":"3163d23bbc8a7f3df0f2d26c81adc358aa3f6a0a33cdb45bb362f567150ce2e5"}
Jul 22 02:48:33.081762 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:33.081669    2565 scope.go:117] "RemoveContainer" containerID="3163d23bbc8a7f3df0f2d26c81adc358aa3f6a0a33cdb45bb362f567150ce2e5"
Jul 22 02:48:33.082779 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:33.082748    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:48:33.082843 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:33.082819    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:48:33.082904 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:33.082872    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:48:34.014785 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:34.014750    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:34.088187 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:34.088156    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w" event={"ID":"3d213f17-a47f-406e-b444-b96a99215261","Type":"ContainerStarted","Data":"1f27b68686d84742eed1ec1d4dd0b73da505e3e1925cbe209b362e788f3dbde7"}
Jul 22 02:48:35.035581 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:35.035535    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:35.037091 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:35.037060    2565 patch_prober.go:28] interesting pod/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 22 02:48:35.037319 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:35.037291    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w" podUID="3d213f17-a47f-406e-b444-b96a99215261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 22 02:48:36.035443 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:36.035403    2565 patch_prober.go:28] interesting pod/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 22 02:48:36.036035 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:36.036001    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w" podUID="3d213f17-a47f-406e-b444-b96a99215261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 22 02:48:37.035644 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:37.035607    2565 patch_prober.go:28] interesting pod/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 22 02:48:37.036320 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:37.036286    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w" podUID="3d213f17-a47f-406e-b444-b96a99215261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 22 02:48:38.035710 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:38.035662    2565 patch_prober.go:28] interesting pod/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 22 02:48:38.036237 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:38.035759    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w" podUID="3d213f17-a47f-406e-b444-b96a99215261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 22 02:48:39.038793 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:39.038761    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:39.039185 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:39.039162    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:39.039691 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:39.039673    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:48:49.454805 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:49.454771    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:48:55.635028 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:55.634990    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"]
Jul 22 02:48:55.635427 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:55.635248    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq" podUID="c59b70d3-4b1f-448c-83de-9010afd3ab7c" containerName="maas-api" containerID="cri-o://56c2bcffbf5d027ed56c4b81b76ad1548f7ba1db27a118d39a9e9d433c105aeb" gracePeriod=30
Jul 22 02:48:55.673440 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:55.673417    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:48:55.884262 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:55.884238    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"
Jul 22 02:48:55.950007 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:55.949938    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59b70d3-4b1f-448c-83de-9010afd3ab7c-openshift-service-ca\") pod \"c59b70d3-4b1f-448c-83de-9010afd3ab7c\" (UID: \"c59b70d3-4b1f-448c-83de-9010afd3ab7c\") "
Jul 22 02:48:55.950007 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:55.949992    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c59b70d3-4b1f-448c-83de-9010afd3ab7c-maas-api-tls\") pod \"c59b70d3-4b1f-448c-83de-9010afd3ab7c\" (UID: \"c59b70d3-4b1f-448c-83de-9010afd3ab7c\") "
Jul 22 02:48:55.950226 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:55.950024    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txbzq\" (UniqueName: \"kubernetes.io/projected/c59b70d3-4b1f-448c-83de-9010afd3ab7c-kube-api-access-txbzq\") pod \"c59b70d3-4b1f-448c-83de-9010afd3ab7c\" (UID: \"c59b70d3-4b1f-448c-83de-9010afd3ab7c\") "
Jul 22 02:48:55.950295 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:55.950252    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c59b70d3-4b1f-448c-83de-9010afd3ab7c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c59b70d3-4b1f-448c-83de-9010afd3ab7c" (UID: "c59b70d3-4b1f-448c-83de-9010afd3ab7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:48:55.952221 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:55.952190    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c59b70d3-4b1f-448c-83de-9010afd3ab7c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c59b70d3-4b1f-448c-83de-9010afd3ab7c" (UID: "c59b70d3-4b1f-448c-83de-9010afd3ab7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:48:55.952221 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:55.952191    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59b70d3-4b1f-448c-83de-9010afd3ab7c-kube-api-access-txbzq" (OuterVolumeSpecName: "kube-api-access-txbzq") pod "c59b70d3-4b1f-448c-83de-9010afd3ab7c" (UID: "c59b70d3-4b1f-448c-83de-9010afd3ab7c"). InnerVolumeSpecName "kube-api-access-txbzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:48:56.051226 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:56.051199    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59b70d3-4b1f-448c-83de-9010afd3ab7c-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:48:56.051226 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:56.051226    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c59b70d3-4b1f-448c-83de-9010afd3ab7c-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:48:56.051383 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:56.051235    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txbzq\" (UniqueName: \"kubernetes.io/projected/c59b70d3-4b1f-448c-83de-9010afd3ab7c-kube-api-access-txbzq\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:48:56.187881 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:56.187827    2565 generic.go:356] "Generic (PLEG): container finished" podID="c59b70d3-4b1f-448c-83de-9010afd3ab7c" containerID="56c2bcffbf5d027ed56c4b81b76ad1548f7ba1db27a118d39a9e9d433c105aeb" exitCode=0
Jul 22 02:48:56.188064 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:56.187901    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"
Jul 22 02:48:56.188064 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:56.187912    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq" event={"ID":"c59b70d3-4b1f-448c-83de-9010afd3ab7c","Type":"ContainerDied","Data":"56c2bcffbf5d027ed56c4b81b76ad1548f7ba1db27a118d39a9e9d433c105aeb"}
Jul 22 02:48:56.188064 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:56.187963    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq" event={"ID":"c59b70d3-4b1f-448c-83de-9010afd3ab7c","Type":"ContainerDied","Data":"ce0d3812e12c81756b8c70150a6d6509ca4bcba4ecc9f73faf3048fea810ad06"}
Jul 22 02:48:56.188064 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:56.187985    2565 scope.go:117] "RemoveContainer" containerID="56c2bcffbf5d027ed56c4b81b76ad1548f7ba1db27a118d39a9e9d433c105aeb"
Jul 22 02:48:56.199613 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:56.199587    2565 scope.go:117] "RemoveContainer" containerID="56c2bcffbf5d027ed56c4b81b76ad1548f7ba1db27a118d39a9e9d433c105aeb"
Jul 22 02:48:56.200215 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:48:56.200163    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c2bcffbf5d027ed56c4b81b76ad1548f7ba1db27a118d39a9e9d433c105aeb\": container with ID starting with 56c2bcffbf5d027ed56c4b81b76ad1548f7ba1db27a118d39a9e9d433c105aeb not found: ID does not exist" containerID="56c2bcffbf5d027ed56c4b81b76ad1548f7ba1db27a118d39a9e9d433c105aeb"
Jul 22 02:48:56.200215 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:56.200199    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c2bcffbf5d027ed56c4b81b76ad1548f7ba1db27a118d39a9e9d433c105aeb"} err="failed to get container status \"56c2bcffbf5d027ed56c4b81b76ad1548f7ba1db27a118d39a9e9d433c105aeb\": rpc error: code = NotFound desc = could not find container \"56c2bcffbf5d027ed56c4b81b76ad1548f7ba1db27a118d39a9e9d433c105aeb\": container with ID starting with 56c2bcffbf5d027ed56c4b81b76ad1548f7ba1db27a118d39a9e9d433c105aeb not found: ID does not exist"
Jul 22 02:48:56.214138 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:56.214114    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"]
Jul 22 02:48:56.218556 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:56.218530    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e3ec0889-7688bc8b69-nb2gq"]
Jul 22 02:48:57.726194 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:48:57.726163    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c59b70d3-4b1f-448c-83de-9010afd3ab7c" path="/var/lib/kubelet/pods/c59b70d3-4b1f-448c-83de-9010afd3ab7c/volumes"
Jul 22 02:49:00.071319 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:00.071285    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:00.206101 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:00.206066    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"]
Jul 22 02:49:00.206363 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:00.206335    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w" podUID="3d213f17-a47f-406e-b444-b96a99215261" containerName="istio-proxy" containerID="cri-o://1f27b68686d84742eed1ec1d4dd0b73da505e3e1925cbe209b362e788f3dbde7" gracePeriod=30
Jul 22 02:49:00.230706 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:00.230680    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:00.463451 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:00.463421    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:00.728100 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:00.728017    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:00.827468 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:00.827434    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:00.947424 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:00.947397    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:01.300296 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.300260    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"]
Jul 22 02:49:01.300661 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.300613    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c59b70d3-4b1f-448c-83de-9010afd3ab7c" containerName="maas-api"
Jul 22 02:49:01.300661 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.300624    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c59b70d3-4b1f-448c-83de-9010afd3ab7c" containerName="maas-api"
Jul 22 02:49:01.300734 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.300697    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c59b70d3-4b1f-448c-83de-9010afd3ab7c" containerName="maas-api"
Jul 22 02:49:01.307102 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.307073    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.310256 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.310233    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-98d2f301-gw-openshift-default-dockercfg-2vngh\""
Jul 22 02:49:01.317494 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.317466    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:01.318447 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.318426    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"]
Jul 22 02:49:01.408226 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.408188    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59ch9\" (UniqueName: \"kubernetes.io/projected/9ecd8230-3172-4819-8c35-61a53ecaf7e2-kube-api-access-59ch9\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.408402 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.408249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istiod-ca-cert\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.408402 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.408337    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-token\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.408402 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.408381    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-podinfo\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.408573 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.408407    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-workload-certs\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.408573 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.408485    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-credential-socket\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.408573 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.408524    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-workload-socket\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.408727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.408590    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-data\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.408727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.408646    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-envoy\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.509901 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.509872    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-data\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.510077 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.509919    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-envoy\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.510077 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.509959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59ch9\" (UniqueName: \"kubernetes.io/projected/9ecd8230-3172-4819-8c35-61a53ecaf7e2-kube-api-access-59ch9\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.510077 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.509987    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istiod-ca-cert\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.510077 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.510026    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-token\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.510283 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.510152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-podinfo\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.510283 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.510227    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-workload-certs\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.510392 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.510283    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-credential-socket\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.510392 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.510294    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-data\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.510392 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.510313    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-workload-socket\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.510600 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.510578    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-workload-socket\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.510689 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.510667    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istiod-ca-cert\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.510730 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.510695    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-credential-socket\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.510768 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.510744    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-workload-certs\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.512324 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.512295    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-envoy\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.512529 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.512503    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-podinfo\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.518802 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.518773    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-token\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.519317 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.519293    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59ch9\" (UniqueName: \"kubernetes.io/projected/9ecd8230-3172-4819-8c35-61a53ecaf7e2-kube-api-access-59ch9\") pod \"e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") " pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.611060 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.611029    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:01.621066 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.621041    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:01.758765 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.758740    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"]
Jul 22 02:49:01.760388 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:49:01.760359    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ecd8230_3172_4819_8c35_61a53ecaf7e2.slice/crio-d5a7a2aadb96054b26f2e0532224ff3b497a2e2dbb3e1dac658b80e888bb4959 WatchSource:0}: Error finding container d5a7a2aadb96054b26f2e0532224ff3b497a2e2dbb3e1dac658b80e888bb4959: Status 404 returned error can't find the container with id d5a7a2aadb96054b26f2e0532224ff3b497a2e2dbb3e1dac658b80e888bb4959
Jul 22 02:49:01.762552 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.762512    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:49:01.762633 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.762596    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:49:01.762680 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:01.762641    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:49:02.220438 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:02.220402    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6" event={"ID":"9ecd8230-3172-4819-8c35-61a53ecaf7e2","Type":"ContainerStarted","Data":"3182d8d5b0739adcfdfa4e21bc78f234f3cff7ed9de816eb2121f88bc92ddc22"}
Jul 22 02:49:02.220638 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:02.220454    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6" event={"ID":"9ecd8230-3172-4819-8c35-61a53ecaf7e2","Type":"ContainerStarted","Data":"d5a7a2aadb96054b26f2e0532224ff3b497a2e2dbb3e1dac658b80e888bb4959"}
Jul 22 02:49:02.251648 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:02.251595    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6" podStartSLOduration=1.2515805420000001 podStartE2EDuration="1.251580542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:49:02.248663069 +0000 UTC m=+1693.096999644" watchObservedRunningTime="2026-07-22 02:49:02.251580542 +0000 UTC m=+1693.099917116"
Jul 22 02:49:02.621930 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:02.621898    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:02.626750 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:02.626725    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:03.226306 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:03.226275    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:03.227293 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:03.227268    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:03.723926 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:03.723887    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:03.821394 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:03.821358    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"]
Jul 22 02:49:03.825015 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:03.824994    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"
Jul 22 02:49:03.827779 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:03.827754    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-98d2f301-serving-cert\""
Jul 22 02:49:03.834203 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:03.834175    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"]
Jul 22 02:49:03.858247 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:03.858213    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:03.932189 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:03.932157    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv92s\" (UniqueName: \"kubernetes.io/projected/4eb2f58f-cf38-4aa8-8196-32f06499b903-kube-api-access-gv92s\") pod \"maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h\" (UID: \"4eb2f58f-cf38-4aa8-8196-32f06499b903\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"
Jul 22 02:49:03.932356 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:03.932213    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb2f58f-cf38-4aa8-8196-32f06499b903-openshift-service-ca\") pod \"maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h\" (UID: \"4eb2f58f-cf38-4aa8-8196-32f06499b903\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"
Jul 22 02:49:03.932400 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:03.932346    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb2f58f-cf38-4aa8-8196-32f06499b903-maas-api-tls\") pod \"maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h\" (UID: \"4eb2f58f-cf38-4aa8-8196-32f06499b903\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"
Jul 22 02:49:04.033540 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:04.033454    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv92s\" (UniqueName: \"kubernetes.io/projected/4eb2f58f-cf38-4aa8-8196-32f06499b903-kube-api-access-gv92s\") pod \"maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h\" (UID: \"4eb2f58f-cf38-4aa8-8196-32f06499b903\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"
Jul 22 02:49:04.033540 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:04.033502    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb2f58f-cf38-4aa8-8196-32f06499b903-openshift-service-ca\") pod \"maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h\" (UID: \"4eb2f58f-cf38-4aa8-8196-32f06499b903\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"
Jul 22 02:49:04.033731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:04.033555    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb2f58f-cf38-4aa8-8196-32f06499b903-maas-api-tls\") pod \"maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h\" (UID: \"4eb2f58f-cf38-4aa8-8196-32f06499b903\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"
Jul 22 02:49:04.033731 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:49:04.033679    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-serving-cert: secret "maas-api-e2e-migrate-98d2f301-serving-cert" not found
Jul 22 02:49:04.033811 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:49:04.033747    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eb2f58f-cf38-4aa8-8196-32f06499b903-maas-api-tls podName:4eb2f58f-cf38-4aa8-8196-32f06499b903 nodeName:}" failed. No retries permitted until 2026-07-22 02:49:04.533725705 +0000 UTC m=+1695.382062256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4eb2f58f-cf38-4aa8-8196-32f06499b903-maas-api-tls") pod "maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h" (UID: "4eb2f58f-cf38-4aa8-8196-32f06499b903") : secret "maas-api-e2e-migrate-98d2f301-serving-cert" not found
Jul 22 02:49:04.034290 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:04.034270    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb2f58f-cf38-4aa8-8196-32f06499b903-openshift-service-ca\") pod \"maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h\" (UID: \"4eb2f58f-cf38-4aa8-8196-32f06499b903\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"
Jul 22 02:49:04.042473 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:04.042437    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv92s\" (UniqueName: \"kubernetes.io/projected/4eb2f58f-cf38-4aa8-8196-32f06499b903-kube-api-access-gv92s\") pod \"maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h\" (UID: \"4eb2f58f-cf38-4aa8-8196-32f06499b903\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"
Jul 22 02:49:04.150761 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:04.150734    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:04.441120 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:04.441091    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:04.538578 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:04.538552    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb2f58f-cf38-4aa8-8196-32f06499b903-maas-api-tls\") pod \"maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h\" (UID: \"4eb2f58f-cf38-4aa8-8196-32f06499b903\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"
Jul 22 02:49:04.541105 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:04.541083    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb2f58f-cf38-4aa8-8196-32f06499b903-maas-api-tls\") pod \"maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h\" (UID: \"4eb2f58f-cf38-4aa8-8196-32f06499b903\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"
Jul 22 02:49:04.541358 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:04.541327    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:04.737871 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:04.737778    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"
Jul 22 02:49:04.882679 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:04.878843    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"]
Jul 22 02:49:05.239446 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.239397    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h" event={"ID":"4eb2f58f-cf38-4aa8-8196-32f06499b903","Type":"ContainerStarted","Data":"8f3d664275afd307569c8ec619e2ef0f187e068b486853974d0fd0f74c1fad25"}
Jul 22 02:49:05.469076 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.469018    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:49:05.547548 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.547520    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d213f17-a47f-406e-b444-b96a99215261-istiod-ca-cert\") pod \"3d213f17-a47f-406e-b444-b96a99215261\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") "
Jul 22 02:49:05.547723 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.547563    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-credential-socket\") pod \"3d213f17-a47f-406e-b444-b96a99215261\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") "
Jul 22 02:49:05.547723 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.547598    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fjct\" (UniqueName: \"kubernetes.io/projected/3d213f17-a47f-406e-b444-b96a99215261-kube-api-access-6fjct\") pod \"3d213f17-a47f-406e-b444-b96a99215261\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") "
Jul 22 02:49:05.547723 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.547654    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-workload-certs\") pod \"3d213f17-a47f-406e-b444-b96a99215261\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") "
Jul 22 02:49:05.547723 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.547687    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d213f17-a47f-406e-b444-b96a99215261-istio-podinfo\") pod \"3d213f17-a47f-406e-b444-b96a99215261\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") "
Jul 22 02:49:05.547977 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.547732    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-workload-socket\") pod \"3d213f17-a47f-406e-b444-b96a99215261\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") "
Jul 22 02:49:05.547977 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.547757    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-istio-data\") pod \"3d213f17-a47f-406e-b444-b96a99215261\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") "
Jul 22 02:49:05.547977 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.547797    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3d213f17-a47f-406e-b444-b96a99215261" (UID: "3d213f17-a47f-406e-b444-b96a99215261"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:49:05.547977 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.547837    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-istio-envoy\") pod \"3d213f17-a47f-406e-b444-b96a99215261\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") "
Jul 22 02:49:05.547977 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.547877    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d213f17-a47f-406e-b444-b96a99215261-istio-token\") pod \"3d213f17-a47f-406e-b444-b96a99215261\" (UID: \"3d213f17-a47f-406e-b444-b96a99215261\") "
Jul 22 02:49:05.548240 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.548054    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3d213f17-a47f-406e-b444-b96a99215261" (UID: "3d213f17-a47f-406e-b444-b96a99215261"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:49:05.548240 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.548087    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d213f17-a47f-406e-b444-b96a99215261-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3d213f17-a47f-406e-b444-b96a99215261" (UID: "3d213f17-a47f-406e-b444-b96a99215261"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:49:05.548240 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.548215    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-workload-socket\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:05.548240 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.548237    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d213f17-a47f-406e-b444-b96a99215261-istiod-ca-cert\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:05.548440 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.548244    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3d213f17-a47f-406e-b444-b96a99215261" (UID: "3d213f17-a47f-406e-b444-b96a99215261"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:49:05.548440 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.548251    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-credential-socket\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:05.548563 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.548540    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-istio-data" (OuterVolumeSpecName: "istio-data") pod "3d213f17-a47f-406e-b444-b96a99215261" (UID: "3d213f17-a47f-406e-b444-b96a99215261"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:49:05.550021 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.549992    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d213f17-a47f-406e-b444-b96a99215261-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3d213f17-a47f-406e-b444-b96a99215261" (UID: "3d213f17-a47f-406e-b444-b96a99215261"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 22 02:49:05.550021 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.550013    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d213f17-a47f-406e-b444-b96a99215261-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d213f17-a47f-406e-b444-b96a99215261" (UID: "3d213f17-a47f-406e-b444-b96a99215261"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:49:05.550330 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.550305    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d213f17-a47f-406e-b444-b96a99215261-kube-api-access-6fjct" (OuterVolumeSpecName: "kube-api-access-6fjct") pod "3d213f17-a47f-406e-b444-b96a99215261" (UID: "3d213f17-a47f-406e-b444-b96a99215261"). InnerVolumeSpecName "kube-api-access-6fjct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:49:05.550424 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.550300    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3d213f17-a47f-406e-b444-b96a99215261" (UID: "3d213f17-a47f-406e-b444-b96a99215261"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:49:05.649771 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.649730    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-workload-certs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:05.649771 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.649770    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d213f17-a47f-406e-b444-b96a99215261-istio-podinfo\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:05.649985 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.649785    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-istio-data\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:05.649985 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.649798    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d213f17-a47f-406e-b444-b96a99215261-istio-envoy\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:05.649985 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.649810    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d213f17-a47f-406e-b444-b96a99215261-istio-token\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:05.649985 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:05.649823    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fjct\" (UniqueName: \"kubernetes.io/projected/3d213f17-a47f-406e-b444-b96a99215261-kube-api-access-6fjct\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:06.245633 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:06.245602    2565 generic.go:356] "Generic (PLEG): container finished" podID="3d213f17-a47f-406e-b444-b96a99215261" containerID="1f27b68686d84742eed1ec1d4dd0b73da505e3e1925cbe209b362e788f3dbde7" exitCode=0
Jul 22 02:49:06.246113 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:06.245653    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"
Jul 22 02:49:06.246113 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:06.245689    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w" event={"ID":"3d213f17-a47f-406e-b444-b96a99215261","Type":"ContainerDied","Data":"1f27b68686d84742eed1ec1d4dd0b73da505e3e1925cbe209b362e788f3dbde7"}
Jul 22 02:49:06.246113 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:06.245749    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w" event={"ID":"3d213f17-a47f-406e-b444-b96a99215261","Type":"ContainerDied","Data":"c2844909ba373640994d89468dd1c1293b6217e0dc744bbcc3b452d7ef7db341"}
Jul 22 02:49:06.246113 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:06.245782    2565 scope.go:117] "RemoveContainer" containerID="1f27b68686d84742eed1ec1d4dd0b73da505e3e1925cbe209b362e788f3dbde7"
Jul 22 02:49:06.247207 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:06.247183    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h" event={"ID":"4eb2f58f-cf38-4aa8-8196-32f06499b903","Type":"ContainerStarted","Data":"58d14ee9b2b730fbcf766da7dfc393d0f47c81c6fd23930d9ed4f2497bea1557"}
Jul 22 02:49:06.247348 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:06.247327    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"
Jul 22 02:49:06.254561 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:06.254545    2565 scope.go:117] "RemoveContainer" containerID="3163d23bbc8a7f3df0f2d26c81adc358aa3f6a0a33cdb45bb362f567150ce2e5"
Jul 22 02:49:06.261936 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:06.261919    2565 scope.go:117] "RemoveContainer" containerID="1f27b68686d84742eed1ec1d4dd0b73da505e3e1925cbe209b362e788f3dbde7"
Jul 22 02:49:06.262182 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:49:06.262163    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f27b68686d84742eed1ec1d4dd0b73da505e3e1925cbe209b362e788f3dbde7\": container with ID starting with 1f27b68686d84742eed1ec1d4dd0b73da505e3e1925cbe209b362e788f3dbde7 not found: ID does not exist" containerID="1f27b68686d84742eed1ec1d4dd0b73da505e3e1925cbe209b362e788f3dbde7"
Jul 22 02:49:06.262237 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:06.262191    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f27b68686d84742eed1ec1d4dd0b73da505e3e1925cbe209b362e788f3dbde7"} err="failed to get container status \"1f27b68686d84742eed1ec1d4dd0b73da505e3e1925cbe209b362e788f3dbde7\": rpc error: code = NotFound desc = could not find container \"1f27b68686d84742eed1ec1d4dd0b73da505e3e1925cbe209b362e788f3dbde7\": container with ID starting with 1f27b68686d84742eed1ec1d4dd0b73da505e3e1925cbe209b362e788f3dbde7 not found: ID does not exist"
Jul 22 02:49:06.262237 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:06.262206    2565 scope.go:117] "RemoveContainer" containerID="3163d23bbc8a7f3df0f2d26c81adc358aa3f6a0a33cdb45bb362f567150ce2e5"
Jul 22 02:49:06.262420 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:49:06.262406    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3163d23bbc8a7f3df0f2d26c81adc358aa3f6a0a33cdb45bb362f567150ce2e5\": container with ID starting with 3163d23bbc8a7f3df0f2d26c81adc358aa3f6a0a33cdb45bb362f567150ce2e5 not found: ID does not exist" containerID="3163d23bbc8a7f3df0f2d26c81adc358aa3f6a0a33cdb45bb362f567150ce2e5"
Jul 22 02:49:06.262466 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:06.262424    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3163d23bbc8a7f3df0f2d26c81adc358aa3f6a0a33cdb45bb362f567150ce2e5"} err="failed to get container status \"3163d23bbc8a7f3df0f2d26c81adc358aa3f6a0a33cdb45bb362f567150ce2e5\": rpc error: code = NotFound desc = could not find container \"3163d23bbc8a7f3df0f2d26c81adc358aa3f6a0a33cdb45bb362f567150ce2e5\": container with ID starting with 3163d23bbc8a7f3df0f2d26c81adc358aa3f6a0a33cdb45bb362f567150ce2e5 not found: ID does not exist"
Jul 22 02:49:06.268307 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:06.268267    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h" podStartSLOduration=2.91731394 podStartE2EDuration="3.268255718s" totalImagesPullingTime="350.941778ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:49:06.265966428 +0000 UTC m=+1697.114303001" watchObservedRunningTime="2026-07-22 02:49:06.268255718 +0000 UTC m=+1697.116592337"
Jul 22 02:49:06.281736 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:06.281717    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"]
Jul 22 02:49:06.284638 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:06.284619    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e3ec0889-openshift-default-74dc88f885-5977w"]
Jul 22 02:49:07.725313 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:07.725279    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d213f17-a47f-406e-b444-b96a99215261" path="/var/lib/kubelet/pods/3d213f17-a47f-406e-b444-b96a99215261/volumes"
Jul 22 02:49:12.258035 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:12.257959    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"
Jul 22 02:49:17.298379 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:17.298347    2565 generic.go:356] "Generic (PLEG): container finished" podID="9ecd8230-3172-4819-8c35-61a53ecaf7e2" containerID="3182d8d5b0739adcfdfa4e21bc78f234f3cff7ed9de816eb2121f88bc92ddc22" exitCode=0
Jul 22 02:49:17.298915 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:17.298420    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6" event={"ID":"9ecd8230-3172-4819-8c35-61a53ecaf7e2","Type":"ContainerDied","Data":"3182d8d5b0739adcfdfa4e21bc78f234f3cff7ed9de816eb2121f88bc92ddc22"}
Jul 22 02:49:17.298990 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:17.298949    2565 scope.go:117] "RemoveContainer" containerID="3182d8d5b0739adcfdfa4e21bc78f234f3cff7ed9de816eb2121f88bc92ddc22"
Jul 22 02:49:17.299940 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:17.299911    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:49:17.300042 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:17.299983    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:49:17.300042 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:17.300015    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:49:18.226419 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:18.226358    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:18.305076 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:18.305043    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6" event={"ID":"9ecd8230-3172-4819-8c35-61a53ecaf7e2","Type":"ContainerStarted","Data":"198974b8adec1e6aa96574c2817592fc17b0475e109d1bd39b92162fec104fb7"}
Jul 22 02:49:18.622027 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:18.621987    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:18.623547 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:18.623216    2565 patch_prober.go:28] interesting pod/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 22 02:49:18.623547 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:18.623274    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6" podUID="9ecd8230-3172-4819-8c35-61a53ecaf7e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 22 02:49:19.622327 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:19.622290    2565 patch_prober.go:28] interesting pod/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 22 02:49:19.622974 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:19.622941    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6" podUID="9ecd8230-3172-4819-8c35-61a53ecaf7e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 22 02:49:20.621911 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:20.621874    2565 patch_prober.go:28] interesting pod/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 22 02:49:20.622204 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:20.622174    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6" podUID="9ecd8230-3172-4819-8c35-61a53ecaf7e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 22 02:49:21.621617 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:21.621579    2565 patch_prober.go:28] interesting pod/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 22 02:49:21.622208 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:21.622177    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6" podUID="9ecd8230-3172-4819-8c35-61a53ecaf7e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 22 02:49:22.625199 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:22.625168    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:22.625567 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:22.625454    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:22.626133 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:22.626115    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:39.243833 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:39.243801    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"]
Jul 22 02:49:39.244405 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:39.244127    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h" podUID="4eb2f58f-cf38-4aa8-8196-32f06499b903" containerName="maas-api" containerID="cri-o://58d14ee9b2b730fbcf766da7dfc393d0f47c81c6fd23930d9ed4f2497bea1557" gracePeriod=30
Jul 22 02:49:39.273818 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:39.273797    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:39.408600 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:39.408567    2565 generic.go:356] "Generic (PLEG): container finished" podID="4eb2f58f-cf38-4aa8-8196-32f06499b903" containerID="58d14ee9b2b730fbcf766da7dfc393d0f47c81c6fd23930d9ed4f2497bea1557" exitCode=0
Jul 22 02:49:39.408600 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:39.408596    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h" event={"ID":"4eb2f58f-cf38-4aa8-8196-32f06499b903","Type":"ContainerDied","Data":"58d14ee9b2b730fbcf766da7dfc393d0f47c81c6fd23930d9ed4f2497bea1557"}
Jul 22 02:49:39.499371 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:39.499313    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"
Jul 22 02:49:39.541522 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:39.541497    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb2f58f-cf38-4aa8-8196-32f06499b903-maas-api-tls\") pod \"4eb2f58f-cf38-4aa8-8196-32f06499b903\" (UID: \"4eb2f58f-cf38-4aa8-8196-32f06499b903\") "
Jul 22 02:49:39.541674 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:39.541541    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gv92s\" (UniqueName: \"kubernetes.io/projected/4eb2f58f-cf38-4aa8-8196-32f06499b903-kube-api-access-gv92s\") pod \"4eb2f58f-cf38-4aa8-8196-32f06499b903\" (UID: \"4eb2f58f-cf38-4aa8-8196-32f06499b903\") "
Jul 22 02:49:39.541674 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:39.541568    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb2f58f-cf38-4aa8-8196-32f06499b903-openshift-service-ca\") pod \"4eb2f58f-cf38-4aa8-8196-32f06499b903\" (UID: \"4eb2f58f-cf38-4aa8-8196-32f06499b903\") "
Jul 22 02:49:39.541959 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:39.541933    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb2f58f-cf38-4aa8-8196-32f06499b903-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4eb2f58f-cf38-4aa8-8196-32f06499b903" (UID: "4eb2f58f-cf38-4aa8-8196-32f06499b903"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:49:39.543641 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:39.543614    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb2f58f-cf38-4aa8-8196-32f06499b903-kube-api-access-gv92s" (OuterVolumeSpecName: "kube-api-access-gv92s") pod "4eb2f58f-cf38-4aa8-8196-32f06499b903" (UID: "4eb2f58f-cf38-4aa8-8196-32f06499b903"). InnerVolumeSpecName "kube-api-access-gv92s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:49:39.543713 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:39.543642    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb2f58f-cf38-4aa8-8196-32f06499b903-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4eb2f58f-cf38-4aa8-8196-32f06499b903" (UID: "4eb2f58f-cf38-4aa8-8196-32f06499b903"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:49:39.642539 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:39.642510    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb2f58f-cf38-4aa8-8196-32f06499b903-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:39.642539 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:39.642537    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv92s\" (UniqueName: \"kubernetes.io/projected/4eb2f58f-cf38-4aa8-8196-32f06499b903-kube-api-access-gv92s\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:39.642680 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:39.642547    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb2f58f-cf38-4aa8-8196-32f06499b903-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:40.414422 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:40.414396    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"
Jul 22 02:49:40.414422 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:40.414415    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h" event={"ID":"4eb2f58f-cf38-4aa8-8196-32f06499b903","Type":"ContainerDied","Data":"8f3d664275afd307569c8ec619e2ef0f187e068b486853974d0fd0f74c1fad25"}
Jul 22 02:49:40.414874 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:40.414470    2565 scope.go:117] "RemoveContainer" containerID="58d14ee9b2b730fbcf766da7dfc393d0f47c81c6fd23930d9ed4f2497bea1557"
Jul 22 02:49:40.432743 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:40.432716    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"]
Jul 22 02:49:40.437402 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:40.437370    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-98d2f301-5744d5cccd-8cl6h"]
Jul 22 02:49:41.725314 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:41.725274    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eb2f58f-cf38-4aa8-8196-32f06499b903" path="/var/lib/kubelet/pods/4eb2f58f-cf38-4aa8-8196-32f06499b903/volumes"
Jul 22 02:49:43.828596 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:43.828566    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:43.970220 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:43.970179    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"]
Jul 22 02:49:43.970429 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:43.970404    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6" podUID="9ecd8230-3172-4819-8c35-61a53ecaf7e2" containerName="istio-proxy" containerID="cri-o://198974b8adec1e6aa96574c2817592fc17b0475e109d1bd39b92162fec104fb7" gracePeriod=30
Jul 22 02:49:43.996894 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:43.996871    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:44.234001 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.233970    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:44.488452 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.488378    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:44.779270 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.779186    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"]
Jul 22 02:49:44.779586 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.779570    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eb2f58f-cf38-4aa8-8196-32f06499b903" containerName="maas-api"
Jul 22 02:49:44.779681 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.779586    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eb2f58f-cf38-4aa8-8196-32f06499b903" containerName="maas-api"
Jul 22 02:49:44.779681 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.779605    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d213f17-a47f-406e-b444-b96a99215261" containerName="istio-proxy"
Jul 22 02:49:44.779681 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.779613    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d213f17-a47f-406e-b444-b96a99215261" containerName="istio-proxy"
Jul 22 02:49:44.779681 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.779620    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d213f17-a47f-406e-b444-b96a99215261" containerName="istio-proxy"
Jul 22 02:49:44.779681 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.779626    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d213f17-a47f-406e-b444-b96a99215261" containerName="istio-proxy"
Jul 22 02:49:44.780014 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.779690    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eb2f58f-cf38-4aa8-8196-32f06499b903" containerName="maas-api"
Jul 22 02:49:44.780014 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.779700    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d213f17-a47f-406e-b444-b96a99215261" containerName="istio-proxy"
Jul 22 02:49:44.780014 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.779707    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d213f17-a47f-406e-b444-b96a99215261" containerName="istio-proxy"
Jul 22 02:49:44.783978 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.783955    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.786626 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.786603    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bef3febf-openshift-default-dockercfg-hl27s\""
Jul 22 02:49:44.794578 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.794554    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"]
Jul 22 02:49:44.814083 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.813833    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:44.896203 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.894734    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a45397d-94fe-447d-b3bf-d2c54f022516-istiod-ca-cert\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.896203 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.894807    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-podinfo\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.896203 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.894895    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-envoy\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.896203 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.894965    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-credential-socket\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.896203 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.894994    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-data\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.896203 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.895044    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djtr4\" (UniqueName: \"kubernetes.io/projected/3a45397d-94fe-447d-b3bf-d2c54f022516-kube-api-access-djtr4\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.896203 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.895096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-token\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.896203 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.895157    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-workload-certs\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.896203 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.895192    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-workload-socket\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.996769 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.996734    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a45397d-94fe-447d-b3bf-d2c54f022516-istiod-ca-cert\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.996769 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.996775    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-podinfo\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.997018 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.996814    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-envoy\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.997018 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.996836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-credential-socket\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.997018 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.996867    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-data\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.997167 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.997024    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djtr4\" (UniqueName: \"kubernetes.io/projected/3a45397d-94fe-447d-b3bf-d2c54f022516-kube-api-access-djtr4\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.997167 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.997087    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-token\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.997167 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.997150    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-workload-certs\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.997327 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.997180    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-workload-socket\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.997327 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.997246    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-data\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.997436 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.997321    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-credential-socket\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.997536 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.997517    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-workload-socket\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.997603 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.997553    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a45397d-94fe-447d-b3bf-d2c54f022516-istiod-ca-cert\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.997603 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.997584    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-workload-certs\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.999381 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.999357    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-envoy\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:44.999471 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:44.999382    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-podinfo\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:45.005730 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:45.005706    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-token\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:45.006050 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:45.006032    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djtr4\" (UniqueName: \"kubernetes.io/projected/3a45397d-94fe-447d-b3bf-d2c54f022516-kube-api-access-djtr4\") pod \"e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") " pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:45.095996 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:45.095907    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:45.229761 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:45.229735    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"]
Jul 22 02:49:45.232075 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:49:45.232045    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a45397d_94fe_447d_b3bf_d2c54f022516.slice/crio-606b00a16e515495e7bbaa8fa16a8c46ce514d50f9b5044829bb5e81b594d7d1 WatchSource:0}: Error finding container 606b00a16e515495e7bbaa8fa16a8c46ce514d50f9b5044829bb5e81b594d7d1: Status 404 returned error can't find the container with id 606b00a16e515495e7bbaa8fa16a8c46ce514d50f9b5044829bb5e81b594d7d1
Jul 22 02:49:45.234243 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:45.234193    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:49:45.234324 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:45.234291    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:49:45.234363 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:45.234340    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:49:45.367030 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:45.366993    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:45.442133 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:45.442097    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7" event={"ID":"3a45397d-94fe-447d-b3bf-d2c54f022516","Type":"ContainerStarted","Data":"e34e975a96051df034eec85919e35c42e57e025c52ab82f08f47254114696089"}
Jul 22 02:49:45.442322 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:45.442156    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7" event={"ID":"3a45397d-94fe-447d-b3bf-d2c54f022516","Type":"ContainerStarted","Data":"606b00a16e515495e7bbaa8fa16a8c46ce514d50f9b5044829bb5e81b594d7d1"}
Jul 22 02:49:45.472023 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:45.471969    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7" podStartSLOduration=1.471951113 podStartE2EDuration="1.471951113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:49:45.471039188 +0000 UTC m=+1736.319375761" watchObservedRunningTime="2026-07-22 02:49:45.471951113 +0000 UTC m=+1736.320287687"
Jul 22 02:49:46.096680 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:46.096650    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:46.101431 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:46.101407    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:46.447371 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:46.447340    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:46.448360 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:46.448338    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:49:46.930592 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:46.930557    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:47.027563 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:47.027535    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:47.127522 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:47.127407    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:47.578975 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:47.578943    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:47.682783 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:47.682756    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:47.778479 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:47.778449    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:49.215803 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.215769    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:49.339908 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.339817    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-podinfo\") pod \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") "
Jul 22 02:49:49.339908 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.339866    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-workload-socket\") pod \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") "
Jul 22 02:49:49.339908 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.339886    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-data\") pod \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") "
Jul 22 02:49:49.340172 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.339952    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59ch9\" (UniqueName: \"kubernetes.io/projected/9ecd8230-3172-4819-8c35-61a53ecaf7e2-kube-api-access-59ch9\") pod \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") "
Jul 22 02:49:49.340172 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.339982    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-token\") pod \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") "
Jul 22 02:49:49.340172 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.339999    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-envoy\") pod \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") "
Jul 22 02:49:49.340172 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.340108    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9ecd8230-3172-4819-8c35-61a53ecaf7e2" (UID: "9ecd8230-3172-4819-8c35-61a53ecaf7e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:49:49.340172 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.340131    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-credential-socket\") pod \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") "
Jul 22 02:49:49.340172 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.340167    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istiod-ca-cert\") pod \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") "
Jul 22 02:49:49.340482 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.340209    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-workload-certs\") pod \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\" (UID: \"9ecd8230-3172-4819-8c35-61a53ecaf7e2\") "
Jul 22 02:49:49.340482 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.340342    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9ecd8230-3172-4819-8c35-61a53ecaf7e2" (UID: "9ecd8230-3172-4819-8c35-61a53ecaf7e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:49:49.340595 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.340551    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-credential-socket\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:49.340595 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.340569    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-workload-socket\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:49.340595 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.340583    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9ecd8230-3172-4819-8c35-61a53ecaf7e2" (UID: "9ecd8230-3172-4819-8c35-61a53ecaf7e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:49:49.340737 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.340596    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9ecd8230-3172-4819-8c35-61a53ecaf7e2" (UID: "9ecd8230-3172-4819-8c35-61a53ecaf7e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:49:49.340883 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.340832    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-data" (OuterVolumeSpecName: "istio-data") pod "9ecd8230-3172-4819-8c35-61a53ecaf7e2" (UID: "9ecd8230-3172-4819-8c35-61a53ecaf7e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:49:49.342142 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.342116    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9ecd8230-3172-4819-8c35-61a53ecaf7e2" (UID: "9ecd8230-3172-4819-8c35-61a53ecaf7e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 22 02:49:49.342259 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.342171    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-token" (OuterVolumeSpecName: "istio-token") pod "9ecd8230-3172-4819-8c35-61a53ecaf7e2" (UID: "9ecd8230-3172-4819-8c35-61a53ecaf7e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:49:49.342259 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.342178    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ecd8230-3172-4819-8c35-61a53ecaf7e2-kube-api-access-59ch9" (OuterVolumeSpecName: "kube-api-access-59ch9") pod "9ecd8230-3172-4819-8c35-61a53ecaf7e2" (UID: "9ecd8230-3172-4819-8c35-61a53ecaf7e2"). InnerVolumeSpecName "kube-api-access-59ch9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:49:49.342408 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.342390    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9ecd8230-3172-4819-8c35-61a53ecaf7e2" (UID: "9ecd8230-3172-4819-8c35-61a53ecaf7e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:49:49.441464 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.441430    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istiod-ca-cert\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:49.441464 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.441460    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-workload-certs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:49.441464 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.441471    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-podinfo\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:49.441728 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.441479    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-data\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:49.441728 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.441489    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59ch9\" (UniqueName: \"kubernetes.io/projected/9ecd8230-3172-4819-8c35-61a53ecaf7e2-kube-api-access-59ch9\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:49.441728 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.441497    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-token\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:49.441728 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.441505    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ecd8230-3172-4819-8c35-61a53ecaf7e2-istio-envoy\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:49:49.463936 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.463902    2565 generic.go:356] "Generic (PLEG): container finished" podID="9ecd8230-3172-4819-8c35-61a53ecaf7e2" containerID="198974b8adec1e6aa96574c2817592fc17b0475e109d1bd39b92162fec104fb7" exitCode=0
Jul 22 02:49:49.464144 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.463950    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"
Jul 22 02:49:49.464144 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.463965    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6" event={"ID":"9ecd8230-3172-4819-8c35-61a53ecaf7e2","Type":"ContainerDied","Data":"198974b8adec1e6aa96574c2817592fc17b0475e109d1bd39b92162fec104fb7"}
Jul 22 02:49:49.464144 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.464019    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6" event={"ID":"9ecd8230-3172-4819-8c35-61a53ecaf7e2","Type":"ContainerDied","Data":"d5a7a2aadb96054b26f2e0532224ff3b497a2e2dbb3e1dac658b80e888bb4959"}
Jul 22 02:49:49.464144 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.464048    2565 scope.go:117] "RemoveContainer" containerID="198974b8adec1e6aa96574c2817592fc17b0475e109d1bd39b92162fec104fb7"
Jul 22 02:49:49.475375 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.475353    2565 scope.go:117] "RemoveContainer" containerID="3182d8d5b0739adcfdfa4e21bc78f234f3cff7ed9de816eb2121f88bc92ddc22"
Jul 22 02:49:49.483237 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.483213    2565 scope.go:117] "RemoveContainer" containerID="198974b8adec1e6aa96574c2817592fc17b0475e109d1bd39b92162fec104fb7"
Jul 22 02:49:49.483505 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:49:49.483481    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"198974b8adec1e6aa96574c2817592fc17b0475e109d1bd39b92162fec104fb7\": container with ID starting with 198974b8adec1e6aa96574c2817592fc17b0475e109d1bd39b92162fec104fb7 not found: ID does not exist" containerID="198974b8adec1e6aa96574c2817592fc17b0475e109d1bd39b92162fec104fb7"
Jul 22 02:49:49.483574 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.483515    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198974b8adec1e6aa96574c2817592fc17b0475e109d1bd39b92162fec104fb7"} err="failed to get container status \"198974b8adec1e6aa96574c2817592fc17b0475e109d1bd39b92162fec104fb7\": rpc error: code = NotFound desc = could not find container \"198974b8adec1e6aa96574c2817592fc17b0475e109d1bd39b92162fec104fb7\": container with ID starting with 198974b8adec1e6aa96574c2817592fc17b0475e109d1bd39b92162fec104fb7 not found: ID does not exist"
Jul 22 02:49:49.483574 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.483536    2565 scope.go:117] "RemoveContainer" containerID="3182d8d5b0739adcfdfa4e21bc78f234f3cff7ed9de816eb2121f88bc92ddc22"
Jul 22 02:49:49.483799 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:49:49.483781    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3182d8d5b0739adcfdfa4e21bc78f234f3cff7ed9de816eb2121f88bc92ddc22\": container with ID starting with 3182d8d5b0739adcfdfa4e21bc78f234f3cff7ed9de816eb2121f88bc92ddc22 not found: ID does not exist" containerID="3182d8d5b0739adcfdfa4e21bc78f234f3cff7ed9de816eb2121f88bc92ddc22"
Jul 22 02:49:49.483841 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.483805    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3182d8d5b0739adcfdfa4e21bc78f234f3cff7ed9de816eb2121f88bc92ddc22"} err="failed to get container status \"3182d8d5b0739adcfdfa4e21bc78f234f3cff7ed9de816eb2121f88bc92ddc22\": rpc error: code = NotFound desc = could not find container \"3182d8d5b0739adcfdfa4e21bc78f234f3cff7ed9de816eb2121f88bc92ddc22\": container with ID starting with 3182d8d5b0739adcfdfa4e21bc78f234f3cff7ed9de816eb2121f88bc92ddc22 not found: ID does not exist"
Jul 22 02:49:49.489270 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.489249    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"]
Jul 22 02:49:49.493514 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.493495    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-98d2f301-gw-openshift-default-8698bb9f87-dg4p6"]
Jul 22 02:49:49.726461 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:49.726434    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ecd8230-3172-4819-8c35-61a53ecaf7e2" path="/var/lib/kubelet/pods/9ecd8230-3172-4819-8c35-61a53ecaf7e2/volumes"
Jul 22 02:49:57.430034 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:57.429992    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:49:57.485033 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:49:57.485000    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:50:01.449105 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:01.449068    2565 patch_prober.go:28] interesting pod/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 22 02:50:01.449530 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:01.449145    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7" podUID="3a45397d-94fe-447d-b3bf-d2c54f022516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 22 02:50:01.517394 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:01.517364    2565 generic.go:356] "Generic (PLEG): container finished" podID="3a45397d-94fe-447d-b3bf-d2c54f022516" containerID="e34e975a96051df034eec85919e35c42e57e025c52ab82f08f47254114696089" exitCode=0
Jul 22 02:50:01.517560 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:01.517442    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7" event={"ID":"3a45397d-94fe-447d-b3bf-d2c54f022516","Type":"ContainerDied","Data":"e34e975a96051df034eec85919e35c42e57e025c52ab82f08f47254114696089"}
Jul 22 02:50:01.517901 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:01.517883    2565 scope.go:117] "RemoveContainer" containerID="e34e975a96051df034eec85919e35c42e57e025c52ab82f08f47254114696089"
Jul 22 02:50:01.518890 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:01.518844    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:50:01.518984 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:01.518929    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:50:01.518984 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:01.518981    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:50:02.524331 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:02.524297    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7" event={"ID":"3a45397d-94fe-447d-b3bf-d2c54f022516","Type":"ContainerStarted","Data":"9a23fccb1b425e15400061be463701f663596df53d6d230e145097b5bbd888a8"}
Jul 22 02:50:03.096819 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:03.096783    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:50:03.097041 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:03.096950    2565 patch_prober.go:28] interesting pod/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 22 02:50:03.097041 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:03.097006    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7" podUID="3a45397d-94fe-447d-b3bf-d2c54f022516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 22 02:50:04.097216 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:04.097178    2565 patch_prober.go:28] interesting pod/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 22 02:50:04.097586 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:04.097243    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7" podUID="3a45397d-94fe-447d-b3bf-d2c54f022516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 22 02:50:05.096500 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:05.096466    2565 patch_prober.go:28] interesting pod/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 22 02:50:05.096669 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:05.096517    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7" podUID="3a45397d-94fe-447d-b3bf-d2c54f022516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 22 02:50:06.096518 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:06.096483    2565 patch_prober.go:28] interesting pod/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 22 02:50:06.096909 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:06.096549    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7" podUID="3a45397d-94fe-447d-b3bf-d2c54f022516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 22 02:50:07.099952 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:07.099924    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:50:07.100403 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:07.100298    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:50:07.100906 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:07.100887    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:50:23.242180 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:23.242149    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:50:27.715465 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:27.715434    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:50:28.912888 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:28.912827    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"]
Jul 22 02:50:28.913607 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:28.913568    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7" podUID="3a45397d-94fe-447d-b3bf-d2c54f022516" containerName="istio-proxy" containerID="cri-o://9a23fccb1b425e15400061be463701f663596df53d6d230e145097b5bbd888a8" gracePeriod=30
Jul 22 02:50:28.934142 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:28.934115    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:50:29.169171 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:29.169091    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:50:29.410680 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:29.410653    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:50:29.723882 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:29.723829    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:50:30.301444 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:30.301409    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:50:34.174796 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.174774    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:50:34.207451 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.207395    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-envoy\") pod \"3a45397d-94fe-447d-b3bf-d2c54f022516\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") "
Jul 22 02:50:34.207451 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.207429    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-token\") pod \"3a45397d-94fe-447d-b3bf-d2c54f022516\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") "
Jul 22 02:50:34.207621 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.207458    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-data\") pod \"3a45397d-94fe-447d-b3bf-d2c54f022516\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") "
Jul 22 02:50:34.207621 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.207490    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-workload-certs\") pod \"3a45397d-94fe-447d-b3bf-d2c54f022516\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") "
Jul 22 02:50:34.207621 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.207522    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-podinfo\") pod \"3a45397d-94fe-447d-b3bf-d2c54f022516\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") "
Jul 22 02:50:34.207621 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.207590    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-workload-socket\") pod \"3a45397d-94fe-447d-b3bf-d2c54f022516\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") "
Jul 22 02:50:34.207621 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.207618    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-credential-socket\") pod \"3a45397d-94fe-447d-b3bf-d2c54f022516\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") "
Jul 22 02:50:34.207941 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.207697    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djtr4\" (UniqueName: \"kubernetes.io/projected/3a45397d-94fe-447d-b3bf-d2c54f022516-kube-api-access-djtr4\") pod \"3a45397d-94fe-447d-b3bf-d2c54f022516\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") "
Jul 22 02:50:34.207941 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.207735    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a45397d-94fe-447d-b3bf-d2c54f022516-istiod-ca-cert\") pod \"3a45397d-94fe-447d-b3bf-d2c54f022516\" (UID: \"3a45397d-94fe-447d-b3bf-d2c54f022516\") "
Jul 22 02:50:34.207941 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.207810    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3a45397d-94fe-447d-b3bf-d2c54f022516" (UID: "3a45397d-94fe-447d-b3bf-d2c54f022516"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:50:34.208101 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.207908    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3a45397d-94fe-447d-b3bf-d2c54f022516" (UID: "3a45397d-94fe-447d-b3bf-d2c54f022516"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:50:34.208101 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.207978    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3a45397d-94fe-447d-b3bf-d2c54f022516" (UID: "3a45397d-94fe-447d-b3bf-d2c54f022516"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:50:34.208303 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.208230    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-workload-socket\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:50:34.208303 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.208262    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-credential-socket\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:50:34.208303 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.208277    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-workload-certs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:50:34.208303 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.208231    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a45397d-94fe-447d-b3bf-d2c54f022516-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3a45397d-94fe-447d-b3bf-d2c54f022516" (UID: "3a45397d-94fe-447d-b3bf-d2c54f022516"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:50:34.208651 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.208633    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-data" (OuterVolumeSpecName: "istio-data") pod "3a45397d-94fe-447d-b3bf-d2c54f022516" (UID: "3a45397d-94fe-447d-b3bf-d2c54f022516"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:50:34.210004 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.209979    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3a45397d-94fe-447d-b3bf-d2c54f022516" (UID: "3a45397d-94fe-447d-b3bf-d2c54f022516"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:50:34.210004 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.209998    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a45397d-94fe-447d-b3bf-d2c54f022516-kube-api-access-djtr4" (OuterVolumeSpecName: "kube-api-access-djtr4") pod "3a45397d-94fe-447d-b3bf-d2c54f022516" (UID: "3a45397d-94fe-447d-b3bf-d2c54f022516"). InnerVolumeSpecName "kube-api-access-djtr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:50:34.210136 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.210018    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-token" (OuterVolumeSpecName: "istio-token") pod "3a45397d-94fe-447d-b3bf-d2c54f022516" (UID: "3a45397d-94fe-447d-b3bf-d2c54f022516"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:50:34.210348 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.210326    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3a45397d-94fe-447d-b3bf-d2c54f022516" (UID: "3a45397d-94fe-447d-b3bf-d2c54f022516"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 22 02:50:34.315103 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.310016    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djtr4\" (UniqueName: \"kubernetes.io/projected/3a45397d-94fe-447d-b3bf-d2c54f022516-kube-api-access-djtr4\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:50:34.315103 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.310050    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a45397d-94fe-447d-b3bf-d2c54f022516-istiod-ca-cert\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:50:34.315103 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.310073    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-envoy\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:50:34.315103 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.310086    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-token\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:50:34.315103 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.310099    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-data\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:50:34.315103 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.310111    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a45397d-94fe-447d-b3bf-d2c54f022516-istio-podinfo\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:50:34.677655 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.677613    2565 generic.go:356] "Generic (PLEG): container finished" podID="3a45397d-94fe-447d-b3bf-d2c54f022516" containerID="9a23fccb1b425e15400061be463701f663596df53d6d230e145097b5bbd888a8" exitCode=0
Jul 22 02:50:34.677874 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.677676    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"
Jul 22 02:50:34.677874 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.677704    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7" event={"ID":"3a45397d-94fe-447d-b3bf-d2c54f022516","Type":"ContainerDied","Data":"9a23fccb1b425e15400061be463701f663596df53d6d230e145097b5bbd888a8"}
Jul 22 02:50:34.677874 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.677798    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7" event={"ID":"3a45397d-94fe-447d-b3bf-d2c54f022516","Type":"ContainerDied","Data":"606b00a16e515495e7bbaa8fa16a8c46ce514d50f9b5044829bb5e81b594d7d1"}
Jul 22 02:50:34.677874 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.677828    2565 scope.go:117] "RemoveContainer" containerID="9a23fccb1b425e15400061be463701f663596df53d6d230e145097b5bbd888a8"
Jul 22 02:50:34.686715 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.686692    2565 scope.go:117] "RemoveContainer" containerID="e34e975a96051df034eec85919e35c42e57e025c52ab82f08f47254114696089"
Jul 22 02:50:34.694330 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.694310    2565 scope.go:117] "RemoveContainer" containerID="9a23fccb1b425e15400061be463701f663596df53d6d230e145097b5bbd888a8"
Jul 22 02:50:34.694580 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:50:34.694561    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a23fccb1b425e15400061be463701f663596df53d6d230e145097b5bbd888a8\": container with ID starting with 9a23fccb1b425e15400061be463701f663596df53d6d230e145097b5bbd888a8 not found: ID does not exist" containerID="9a23fccb1b425e15400061be463701f663596df53d6d230e145097b5bbd888a8"
Jul 22 02:50:34.694629 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.694590    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a23fccb1b425e15400061be463701f663596df53d6d230e145097b5bbd888a8"} err="failed to get container status \"9a23fccb1b425e15400061be463701f663596df53d6d230e145097b5bbd888a8\": rpc error: code = NotFound desc = could not find container \"9a23fccb1b425e15400061be463701f663596df53d6d230e145097b5bbd888a8\": container with ID starting with 9a23fccb1b425e15400061be463701f663596df53d6d230e145097b5bbd888a8 not found: ID does not exist"
Jul 22 02:50:34.694629 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.694609    2565 scope.go:117] "RemoveContainer" containerID="e34e975a96051df034eec85919e35c42e57e025c52ab82f08f47254114696089"
Jul 22 02:50:34.694879 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:50:34.694859    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e34e975a96051df034eec85919e35c42e57e025c52ab82f08f47254114696089\": container with ID starting with e34e975a96051df034eec85919e35c42e57e025c52ab82f08f47254114696089 not found: ID does not exist" containerID="e34e975a96051df034eec85919e35c42e57e025c52ab82f08f47254114696089"
Jul 22 02:50:34.694940 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.694883    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34e975a96051df034eec85919e35c42e57e025c52ab82f08f47254114696089"} err="failed to get container status \"e34e975a96051df034eec85919e35c42e57e025c52ab82f08f47254114696089\": rpc error: code = NotFound desc = could not find container \"e34e975a96051df034eec85919e35c42e57e025c52ab82f08f47254114696089\": container with ID starting with e34e975a96051df034eec85919e35c42e57e025c52ab82f08f47254114696089 not found: ID does not exist"
Jul 22 02:50:34.699534 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.699513    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"]
Jul 22 02:50:34.703365 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.703346    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bef3febf-openshift-default-5788bd77f7-xt5q7"]
Jul 22 02:50:34.913609 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:34.913575    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:50:35.014346 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:35.014258    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:50:35.725163 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:35.725133    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a45397d-94fe-447d-b3bf-d2c54f022516" path="/var/lib/kubelet/pods/3a45397d-94fe-447d-b3bf-d2c54f022516/volumes"
Jul 22 02:50:37.410781 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.410749    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:50:37.485140 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.485104    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"]
Jul 22 02:50:37.485469 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.485457    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a45397d-94fe-447d-b3bf-d2c54f022516" containerName="istio-proxy"
Jul 22 02:50:37.485513 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.485471    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a45397d-94fe-447d-b3bf-d2c54f022516" containerName="istio-proxy"
Jul 22 02:50:37.485513 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.485484    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a45397d-94fe-447d-b3bf-d2c54f022516" containerName="istio-proxy"
Jul 22 02:50:37.485513 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.485491    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a45397d-94fe-447d-b3bf-d2c54f022516" containerName="istio-proxy"
Jul 22 02:50:37.485513 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.485499    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ecd8230-3172-4819-8c35-61a53ecaf7e2" containerName="istio-proxy"
Jul 22 02:50:37.485513 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.485505    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ecd8230-3172-4819-8c35-61a53ecaf7e2" containerName="istio-proxy"
Jul 22 02:50:37.486059 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.485518    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ecd8230-3172-4819-8c35-61a53ecaf7e2" containerName="istio-proxy"
Jul 22 02:50:37.486059 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.485524    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ecd8230-3172-4819-8c35-61a53ecaf7e2" containerName="istio-proxy"
Jul 22 02:50:37.486059 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.485577    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a45397d-94fe-447d-b3bf-d2c54f022516" containerName="istio-proxy"
Jul 22 02:50:37.486059 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.485584    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ecd8230-3172-4819-8c35-61a53ecaf7e2" containerName="istio-proxy"
Jul 22 02:50:37.486059 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.485592    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ecd8230-3172-4819-8c35-61a53ecaf7e2" containerName="istio-proxy"
Jul 22 02:50:37.488814 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.488788    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:50:37.488814 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.488805    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"
Jul 22 02:50:37.491162 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.491143    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e7d31a47-serving-cert\""
Jul 22 02:50:37.499427 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.499405    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"]
Jul 22 02:50:37.533612 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.533591    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed87f705-976e-4923-94a9-21033f0efce3-maas-api-tls\") pod \"maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m\" (UID: \"ed87f705-976e-4923-94a9-21033f0efce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"
Jul 22 02:50:37.533734 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.533629    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4x5r\" (UniqueName: \"kubernetes.io/projected/ed87f705-976e-4923-94a9-21033f0efce3-kube-api-access-j4x5r\") pod \"maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m\" (UID: \"ed87f705-976e-4923-94a9-21033f0efce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"
Jul 22 02:50:37.533734 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.533652    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed87f705-976e-4923-94a9-21033f0efce3-openshift-service-ca\") pod \"maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m\" (UID: \"ed87f705-976e-4923-94a9-21033f0efce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"
Jul 22 02:50:37.568540 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.568512    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:50:37.635188 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.635160    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed87f705-976e-4923-94a9-21033f0efce3-maas-api-tls\") pod \"maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m\" (UID: \"ed87f705-976e-4923-94a9-21033f0efce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"
Jul 22 02:50:37.635364 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.635203    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4x5r\" (UniqueName: \"kubernetes.io/projected/ed87f705-976e-4923-94a9-21033f0efce3-kube-api-access-j4x5r\") pod \"maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m\" (UID: \"ed87f705-976e-4923-94a9-21033f0efce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"
Jul 22 02:50:37.635364 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:50:37.635311    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-serving-cert: secret "maas-api-e2e-derive-e7d31a47-serving-cert" not found
Jul 22 02:50:37.635364 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.635337    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed87f705-976e-4923-94a9-21033f0efce3-openshift-service-ca\") pod \"maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m\" (UID: \"ed87f705-976e-4923-94a9-21033f0efce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"
Jul 22 02:50:37.635484 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:50:37.635378    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed87f705-976e-4923-94a9-21033f0efce3-maas-api-tls podName:ed87f705-976e-4923-94a9-21033f0efce3 nodeName:}" failed. No retries permitted until 2026-07-22 02:50:38.135360911 +0000 UTC m=+1788.983697462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed87f705-976e-4923-94a9-21033f0efce3-maas-api-tls") pod "maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m" (UID: "ed87f705-976e-4923-94a9-21033f0efce3") : secret "maas-api-e2e-derive-e7d31a47-serving-cert" not found
Jul 22 02:50:37.636056 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.636037    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed87f705-976e-4923-94a9-21033f0efce3-openshift-service-ca\") pod \"maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m\" (UID: \"ed87f705-976e-4923-94a9-21033f0efce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"
Jul 22 02:50:37.647384 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.647360    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4x5r\" (UniqueName: \"kubernetes.io/projected/ed87f705-976e-4923-94a9-21033f0efce3-kube-api-access-j4x5r\") pod \"maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m\" (UID: \"ed87f705-976e-4923-94a9-21033f0efce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"
Jul 22 02:50:37.771111 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:37.771036    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:50:38.065271 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:38.065195    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:50:38.140133 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:38.140104    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed87f705-976e-4923-94a9-21033f0efce3-maas-api-tls\") pod \"maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m\" (UID: \"ed87f705-976e-4923-94a9-21033f0efce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"
Jul 22 02:50:38.142489 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:38.142469    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed87f705-976e-4923-94a9-21033f0efce3-maas-api-tls\") pod \"maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m\" (UID: \"ed87f705-976e-4923-94a9-21033f0efce3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"
Jul 22 02:50:38.169257 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:38.169233    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:50:38.399431 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:38.399400    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"
Jul 22 02:50:38.532858 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:38.532820    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"]
Jul 22 02:50:38.534406 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:50:38.534380    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded87f705_976e_4923_94a9_21033f0efce3.slice/crio-423f8ace6a13206b2ac0cc64dac8182540eb4f10b10e2161e004b4a2b0e8813a WatchSource:0}: Error finding container 423f8ace6a13206b2ac0cc64dac8182540eb4f10b10e2161e004b4a2b0e8813a: Status 404 returned error can't find the container with id 423f8ace6a13206b2ac0cc64dac8182540eb4f10b10e2161e004b4a2b0e8813a
Jul 22 02:50:38.696844 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:38.696754    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m" event={"ID":"ed87f705-976e-4923-94a9-21033f0efce3","Type":"ContainerStarted","Data":"423f8ace6a13206b2ac0cc64dac8182540eb4f10b10e2161e004b4a2b0e8813a"}
Jul 22 02:50:39.703108 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:39.703075    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m" event={"ID":"ed87f705-976e-4923-94a9-21033f0efce3","Type":"ContainerStarted","Data":"b925089b86a6aae9a15b9c3f6b3c38757b5e435f83290e9ab234c1b52b7ff5b8"}
Jul 22 02:50:39.703531 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:39.703140    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"
Jul 22 02:50:39.730050 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:39.729982    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m" podStartSLOduration=2.410585514 podStartE2EDuration="2.72995905s" totalImagesPullingTime="319.373536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:50:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:50:39.724166724 +0000 UTC m=+1790.572503308" watchObservedRunningTime="2026-07-22 02:50:39.72995905 +0000 UTC m=+1790.578295625"
Jul 22 02:50:45.713610 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:50:45.713536    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"
Jul 22 02:51:02.298654 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:02.298612    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7d31a47-w6s2p"]
Jul 22 02:51:02.299279 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:02.299257    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a45397d-94fe-447d-b3bf-d2c54f022516" containerName="istio-proxy"
Jul 22 02:51:02.302347 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:02.302324    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7d31a47-w6s2p"
Jul 22 02:51:02.305460 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:02.305438    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vtlqd\""
Jul 22 02:51:02.312407 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:02.312381    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7d31a47-w6s2p"]
Jul 22 02:51:02.443434 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:02.443394    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ee042bf-a42a-487e-9ca4-b7a0c09fd362-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7d31a47-w6s2p\" (UID: \"0ee042bf-a42a-487e-9ca4-b7a0c09fd362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7d31a47-w6s2p"
Jul 22 02:51:02.544183 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:02.544151    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ee042bf-a42a-487e-9ca4-b7a0c09fd362-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7d31a47-w6s2p\" (UID: \"0ee042bf-a42a-487e-9ca4-b7a0c09fd362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7d31a47-w6s2p"
Jul 22 02:51:02.544799 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:02.544780    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ee042bf-a42a-487e-9ca4-b7a0c09fd362-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7d31a47-w6s2p\" (UID: \"0ee042bf-a42a-487e-9ca4-b7a0c09fd362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7d31a47-w6s2p"
Jul 22 02:51:02.613885 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:02.613841    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7d31a47-w6s2p"
Jul 22 02:51:02.743502 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:02.743473    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7d31a47-w6s2p"]
Jul 22 02:51:02.745556 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:51:02.745523    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee042bf_a42a_487e_9ca4_b7a0c09fd362.slice/crio-d8c5ec818e7ba36bfd91a1bc235df2f09c8f33084f9d8b707dd2888ed1c5571f WatchSource:0}: Error finding container d8c5ec818e7ba36bfd91a1bc235df2f09c8f33084f9d8b707dd2888ed1c5571f: Status 404 returned error can't find the container with id d8c5ec818e7ba36bfd91a1bc235df2f09c8f33084f9d8b707dd2888ed1c5571f
Jul 22 02:51:02.801926 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:02.801890    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7d31a47-w6s2p" event={"ID":"0ee042bf-a42a-487e-9ca4-b7a0c09fd362","Type":"ContainerStarted","Data":"d8c5ec818e7ba36bfd91a1bc235df2f09c8f33084f9d8b707dd2888ed1c5571f"}
Jul 22 02:51:16.867173 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:16.867145    2565 generic.go:356] "Generic (PLEG): container finished" podID="0ee042bf-a42a-487e-9ca4-b7a0c09fd362" containerID="28acbcfb16816b483d14a66222f5a659dac4c91447986641a2e4b8a7b5b57832" exitCode=0
Jul 22 02:51:16.867463 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:16.867184    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7d31a47-w6s2p" event={"ID":"0ee042bf-a42a-487e-9ca4-b7a0c09fd362","Type":"ContainerDied","Data":"28acbcfb16816b483d14a66222f5a659dac4c91447986641a2e4b8a7b5b57832"}
Jul 22 02:51:18.004492 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:18.004470    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7d31a47-w6s2p"
Jul 22 02:51:18.174806 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:18.174718    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ee042bf-a42a-487e-9ca4-b7a0c09fd362-maas-api-service-ca\") pod \"0ee042bf-a42a-487e-9ca4-b7a0c09fd362\" (UID: \"0ee042bf-a42a-487e-9ca4-b7a0c09fd362\") "
Jul 22 02:51:18.175054 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:18.175031    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee042bf-a42a-487e-9ca4-b7a0c09fd362-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ee042bf-a42a-487e-9ca4-b7a0c09fd362" (UID: "0ee042bf-a42a-487e-9ca4-b7a0c09fd362"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:51:18.276118 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:18.276086    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ee042bf-a42a-487e-9ca4-b7a0c09fd362-maas-api-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:51:18.878897 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:18.878841    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7d31a47-w6s2p"
Jul 22 02:51:18.878897 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:18.878867    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7d31a47-w6s2p" event={"ID":"0ee042bf-a42a-487e-9ca4-b7a0c09fd362","Type":"ContainerDied","Data":"d8c5ec818e7ba36bfd91a1bc235df2f09c8f33084f9d8b707dd2888ed1c5571f"}
Jul 22 02:51:18.879106 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:18.878911    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8c5ec818e7ba36bfd91a1bc235df2f09c8f33084f9d8b707dd2888ed1c5571f"
Jul 22 02:51:22.742757 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:22.742725    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"]
Jul 22 02:51:22.743182 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:22.742955    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m" podUID="ed87f705-976e-4923-94a9-21033f0efce3" containerName="maas-api" containerID="cri-o://b925089b86a6aae9a15b9c3f6b3c38757b5e435f83290e9ab234c1b52b7ff5b8" gracePeriod=30
Jul 22 02:51:22.766298 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:22.766270    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:51:22.897784 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:22.897752    2565 generic.go:356] "Generic (PLEG): container finished" podID="ed87f705-976e-4923-94a9-21033f0efce3" containerID="b925089b86a6aae9a15b9c3f6b3c38757b5e435f83290e9ab234c1b52b7ff5b8" exitCode=0
Jul 22 02:51:22.897956 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:22.897822    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m" event={"ID":"ed87f705-976e-4923-94a9-21033f0efce3","Type":"ContainerDied","Data":"b925089b86a6aae9a15b9c3f6b3c38757b5e435f83290e9ab234c1b52b7ff5b8"}
Jul 22 02:51:23.000523 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:23.000471    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"
Jul 22 02:51:23.113391 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:23.113364    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed87f705-976e-4923-94a9-21033f0efce3-openshift-service-ca\") pod \"ed87f705-976e-4923-94a9-21033f0efce3\" (UID: \"ed87f705-976e-4923-94a9-21033f0efce3\") "
Jul 22 02:51:23.113589 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:23.113434    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed87f705-976e-4923-94a9-21033f0efce3-maas-api-tls\") pod \"ed87f705-976e-4923-94a9-21033f0efce3\" (UID: \"ed87f705-976e-4923-94a9-21033f0efce3\") "
Jul 22 02:51:23.113589 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:23.113461    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4x5r\" (UniqueName: \"kubernetes.io/projected/ed87f705-976e-4923-94a9-21033f0efce3-kube-api-access-j4x5r\") pod \"ed87f705-976e-4923-94a9-21033f0efce3\" (UID: \"ed87f705-976e-4923-94a9-21033f0efce3\") "
Jul 22 02:51:23.114010 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:23.113963    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed87f705-976e-4923-94a9-21033f0efce3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ed87f705-976e-4923-94a9-21033f0efce3" (UID: "ed87f705-976e-4923-94a9-21033f0efce3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:51:23.115779 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:23.115750    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed87f705-976e-4923-94a9-21033f0efce3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ed87f705-976e-4923-94a9-21033f0efce3" (UID: "ed87f705-976e-4923-94a9-21033f0efce3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:51:23.115920 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:23.115875    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed87f705-976e-4923-94a9-21033f0efce3-kube-api-access-j4x5r" (OuterVolumeSpecName: "kube-api-access-j4x5r") pod "ed87f705-976e-4923-94a9-21033f0efce3" (UID: "ed87f705-976e-4923-94a9-21033f0efce3"). InnerVolumeSpecName "kube-api-access-j4x5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:51:23.215059 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:23.215034    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed87f705-976e-4923-94a9-21033f0efce3-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:51:23.215059 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:23.215058    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed87f705-976e-4923-94a9-21033f0efce3-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:51:23.215215 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:23.215068    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4x5r\" (UniqueName: \"kubernetes.io/projected/ed87f705-976e-4923-94a9-21033f0efce3-kube-api-access-j4x5r\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:51:23.903465 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:23.903436    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"
Jul 22 02:51:23.903871 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:23.903484    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m" event={"ID":"ed87f705-976e-4923-94a9-21033f0efce3","Type":"ContainerDied","Data":"423f8ace6a13206b2ac0cc64dac8182540eb4f10b10e2161e004b4a2b0e8813a"}
Jul 22 02:51:23.903871 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:23.903540    2565 scope.go:117] "RemoveContainer" containerID="b925089b86a6aae9a15b9c3f6b3c38757b5e435f83290e9ab234c1b52b7ff5b8"
Jul 22 02:51:23.921509 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:23.921486    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"]
Jul 22 02:51:23.925403 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:23.925379    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7d31a47-77d78df689-dqb2m"]
Jul 22 02:51:25.726297 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:25.726263    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed87f705-976e-4923-94a9-21033f0efce3" path="/var/lib/kubelet/pods/ed87f705-976e-4923-94a9-21033f0efce3/volumes"
Jul 22 02:51:27.383138 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:27.383107    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:51:27.456014 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:27.455983    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7d31a47-w6s2p"]
Jul 22 02:51:27.461150 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:27.461118    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7d31a47-w6s2p"]
Jul 22 02:51:27.648331 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:27.648238    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:51:27.726075 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:27.726045    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ee042bf-a42a-487e-9ca4-b7a0c09fd362" path="/var/lib/kubelet/pods/0ee042bf-a42a-487e-9ca4-b7a0c09fd362/volumes"
Jul 22 02:51:27.862210 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:27.862177    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:51:28.378366 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:28.378336    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:51:28.469798 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:28.469768    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:51:28.570624 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:28.570596    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:51:30.131312 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:30.131282    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:51:36.190650 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:36.190617    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:51:36.269047 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:36.269013    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:51:36.386946 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:36.386915    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:51:36.485786 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:36.485705    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:51:36.922553 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:36.922521    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:51:42.479298 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:42.479264    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:51:42.579759 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:51:42.579729    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:52:02.931169 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:52:02.931129    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:52:09.477729 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:52:09.477689    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:52:09.587994 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:52:09.587958    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:52:09.682443 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:52:09.682415    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:52:25.634487 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:52:25.634408    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:52:31.671424 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:52:31.671377    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:52:31.757150 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:52:31.757116    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:52:31.904558 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:52:31.904531    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:52:32.005369 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:52:32.005289    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:52:42.460181 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:52:42.460122    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:52:43.281883 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:52:43.281831    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:52:50.906136 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:52:50.906101    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:52:52.255150 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:52:52.255118    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:52:52.410372 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:52:52.410332    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:52:54.167485 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:52:54.167445    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:00.233642 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:00.233610    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:00.327834 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:00.327804    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:00.960093 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:00.960065    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:08.299602 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.299566    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:08.396235 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.396206    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:08.560653 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.560581    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:08.874910 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.874879    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"]
Jul 22 02:53:08.875245 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.875233    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ee042bf-a42a-487e-9ca4-b7a0c09fd362" containerName="revoke-keys"
Jul 22 02:53:08.875304 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.875247    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ee042bf-a42a-487e-9ca4-b7a0c09fd362" containerName="revoke-keys"
Jul 22 02:53:08.875304 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.875261    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed87f705-976e-4923-94a9-21033f0efce3" containerName="maas-api"
Jul 22 02:53:08.875304 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.875267    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed87f705-976e-4923-94a9-21033f0efce3" containerName="maas-api"
Jul 22 02:53:08.875411 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.875321    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed87f705-976e-4923-94a9-21033f0efce3" containerName="maas-api"
Jul 22 02:53:08.875411 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.875333    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ee042bf-a42a-487e-9ca4-b7a0c09fd362" containerName="revoke-keys"
Jul 22 02:53:08.878205 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.878188    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:08.882269 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.882237    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0a5791b8-openshift-default-dockercfg-v67p7\""
Jul 22 02:53:08.907121 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.907094    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"]
Jul 22 02:53:08.921551 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.921522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-credential-socket\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:08.921673 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.921557    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-workload-certs\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:08.921673 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.921603    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istiod-ca-cert\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:08.921673 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.921669    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-workload-socket\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:08.921822 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.921686    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-envoy\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:08.921822 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.921701    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-podinfo\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:08.921822 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.921720    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-data\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:08.921822 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.921801    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkrp7\" (UniqueName: \"kubernetes.io/projected/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-kube-api-access-jkrp7\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:08.922018 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.921882    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-token\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:08.929267 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:08.929238    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:09.022589 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.022547    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-workload-socket\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:09.022589 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.022590    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-envoy\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:09.022805 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.022614    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-podinfo\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:09.022805 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.022719    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-data\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:09.022805 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.022761    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkrp7\" (UniqueName: \"kubernetes.io/projected/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-kube-api-access-jkrp7\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:09.022805 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.022794    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-token\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:09.023052 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.023037    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-workload-socket\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:09.023275 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.023132    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-data\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:09.023275 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.023139    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-credential-socket\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:09.023275 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.023210    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-workload-certs\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:09.023275 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.023268    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istiod-ca-cert\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:09.023543 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.023385    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-credential-socket\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:09.023634 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.023611    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-workload-certs\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:09.024022 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.024001    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istiod-ca-cert\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:09.025174 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.025149    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-envoy\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:09.025349 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.025326    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-podinfo\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:09.033397 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.033379    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-token\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:09.034137 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.034121    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkrp7\" (UniqueName: \"kubernetes.io/projected/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-kube-api-access-jkrp7\") pod \"e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") " pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:09.189365 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.189284    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:09.262693 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.262662    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:09.329947 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.329920    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"]
Jul 22 02:53:09.331506 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:53:09.331482    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4312d4e_2fb5_4c6a_ab25_39cb47c313c9.slice/crio-b85577cf56aad4733f2e7ceb345c03a16a366ed58325c4a841902c7c2885ba60 WatchSource:0}: Error finding container b85577cf56aad4733f2e7ceb345c03a16a366ed58325c4a841902c7c2885ba60: Status 404 returned error can't find the container with id b85577cf56aad4733f2e7ceb345c03a16a366ed58325c4a841902c7c2885ba60
Jul 22 02:53:09.333526 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.333496    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:53:09.333590 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.333559    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:53:09.333629 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.333587    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:53:09.393182 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:09.393161    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx" event={"ID":"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9","Type":"ContainerStarted","Data":"b85577cf56aad4733f2e7ceb345c03a16a366ed58325c4a841902c7c2885ba60"}
Jul 22 02:53:10.399708 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:10.399668    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx" event={"ID":"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9","Type":"ContainerStarted","Data":"52530c100cdbfbd720da25c8e65a410c9c06efd64a2c166bc6d94835dac0450c"}
Jul 22 02:53:10.424164 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:10.424112    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx" podStartSLOduration=2.424097218 podStartE2EDuration="2.424097218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:53:10.42345249 +0000 UTC m=+1941.271789065" watchObservedRunningTime="2026-07-22 02:53:10.424097218 +0000 UTC m=+1941.272433791"
Jul 22 02:53:11.189798 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:11.189767    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:11.194907 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:11.194878    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:11.404118 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:11.404094    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:11.404962 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:11.404942    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:12.395947 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:12.395897    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:12.502524 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:12.499818    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"]
Jul 22 02:53:12.504702 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:12.504673    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"
Jul 22 02:53:12.506679 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:12.506651    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:12.507541 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:12.507523    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0a5791b8-serving-cert\""
Jul 22 02:53:12.512456 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:12.512432    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"]
Jul 22 02:53:12.556631 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:12.556597    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8snp\" (UniqueName: \"kubernetes.io/projected/05894367-b97b-4f85-bf13-92d002462e1d-kube-api-access-v8snp\") pod \"maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n\" (UID: \"05894367-b97b-4f85-bf13-92d002462e1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"
Jul 22 02:53:12.556765 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:12.556655    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05894367-b97b-4f85-bf13-92d002462e1d-maas-api-tls\") pod \"maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n\" (UID: \"05894367-b97b-4f85-bf13-92d002462e1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"
Jul 22 02:53:12.556765 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:12.556729    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05894367-b97b-4f85-bf13-92d002462e1d-openshift-service-ca\") pod \"maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n\" (UID: \"05894367-b97b-4f85-bf13-92d002462e1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"
Jul 22 02:53:12.584634 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:12.584598    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:12.658173 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:12.658070    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8snp\" (UniqueName: \"kubernetes.io/projected/05894367-b97b-4f85-bf13-92d002462e1d-kube-api-access-v8snp\") pod \"maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n\" (UID: \"05894367-b97b-4f85-bf13-92d002462e1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"
Jul 22 02:53:12.658173 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:12.658149    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05894367-b97b-4f85-bf13-92d002462e1d-maas-api-tls\") pod \"maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n\" (UID: \"05894367-b97b-4f85-bf13-92d002462e1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"
Jul 22 02:53:12.658422 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:12.658187    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05894367-b97b-4f85-bf13-92d002462e1d-openshift-service-ca\") pod \"maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n\" (UID: \"05894367-b97b-4f85-bf13-92d002462e1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"
Jul 22 02:53:12.659312 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:12.659283    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05894367-b97b-4f85-bf13-92d002462e1d-openshift-service-ca\") pod \"maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n\" (UID: \"05894367-b97b-4f85-bf13-92d002462e1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"
Jul 22 02:53:12.661073 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:12.661048    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05894367-b97b-4f85-bf13-92d002462e1d-maas-api-tls\") pod \"maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n\" (UID: \"05894367-b97b-4f85-bf13-92d002462e1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"
Jul 22 02:53:12.669087 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:12.669064    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8snp\" (UniqueName: \"kubernetes.io/projected/05894367-b97b-4f85-bf13-92d002462e1d-kube-api-access-v8snp\") pod \"maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n\" (UID: \"05894367-b97b-4f85-bf13-92d002462e1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"
Jul 22 02:53:12.685890 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:12.685866    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:12.820670 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:12.820638    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"
Jul 22 02:53:12.958925 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:12.958896    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"]
Jul 22 02:53:12.960244 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:53:12.960218    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05894367_b97b_4f85_bf13_92d002462e1d.slice/crio-011fd2fc17948c18b79573ff639648b768ebf05feed5ac8825014c1fc638cf92 WatchSource:0}: Error finding container 011fd2fc17948c18b79573ff639648b768ebf05feed5ac8825014c1fc638cf92: Status 404 returned error can't find the container with id 011fd2fc17948c18b79573ff639648b768ebf05feed5ac8825014c1fc638cf92
Jul 22 02:53:13.127799 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:13.127762    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:13.236443 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:13.236358    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:13.418181 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:13.418142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n" event={"ID":"05894367-b97b-4f85-bf13-92d002462e1d","Type":"ContainerStarted","Data":"d40cde91378e76bd4ac6b80eba267491cdcb700256d5ef6144c81ff42ce082a3"}
Jul 22 02:53:13.418387 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:13.418217    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n" event={"ID":"05894367-b97b-4f85-bf13-92d002462e1d","Type":"ContainerStarted","Data":"011fd2fc17948c18b79573ff639648b768ebf05feed5ac8825014c1fc638cf92"}
Jul 22 02:53:13.418471 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:13.418441    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"
Jul 22 02:53:13.436466 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:13.436396    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n" podStartSLOduration=1.086965732 podStartE2EDuration="1.436376443s" totalImagesPullingTime="349.410711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:53:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:53:13.435587483 +0000 UTC m=+1944.283924058" watchObservedRunningTime="2026-07-22 02:53:13.436376443 +0000 UTC m=+1944.284713018"
Jul 22 02:53:16.924682 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:16.924649    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:17.587241 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:17.587208    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:17.684314 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:17.684283    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:18.052050 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:18.052011    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:19.438226 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:19.438182    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"
Jul 22 02:53:23.988177 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:23.988130    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:24.080989 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:24.080931    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:26.405727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:26.405694    2565 patch_prober.go:28] interesting pod/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 22 02:53:26.406153 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:26.405745    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx" podUID="b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 22 02:53:26.480820 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:26.480744    2565 generic.go:356] "Generic (PLEG): container finished" podID="b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" containerID="52530c100cdbfbd720da25c8e65a410c9c06efd64a2c166bc6d94835dac0450c" exitCode=0
Jul 22 02:53:26.480975 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:26.480817    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx" event={"ID":"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9","Type":"ContainerDied","Data":"52530c100cdbfbd720da25c8e65a410c9c06efd64a2c166bc6d94835dac0450c"}
Jul 22 02:53:26.481240 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:26.481226    2565 scope.go:117] "RemoveContainer" containerID="52530c100cdbfbd720da25c8e65a410c9c06efd64a2c166bc6d94835dac0450c"
Jul 22 02:53:26.482196 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:26.482167    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:53:26.482296 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:26.482258    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:53:26.482363 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:26.482304    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:53:27.488185 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:27.488151    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx" event={"ID":"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9","Type":"ContainerStarted","Data":"e158b426b6a105e633a397a029aa4a4987104bd0534fe0a39aba3fc4596d4e8e"}
Jul 22 02:53:28.190082 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:28.190046    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:28.190343 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:28.190318    2565 patch_prober.go:28] interesting pod/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 22 02:53:28.190400 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:28.190369    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx" podUID="b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 22 02:53:29.190455 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:29.190414    2565 patch_prober.go:28] interesting pod/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 22 02:53:29.190836 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:29.190479    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx" podUID="b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 22 02:53:30.190397 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:30.190363    2565 patch_prober.go:28] interesting pod/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 22 02:53:30.190587 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:30.190418    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx" podUID="b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 22 02:53:31.190290 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:31.190253    2565 patch_prober.go:28] interesting pod/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 22 02:53:31.190471 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:31.190311    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx" podUID="b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 22 02:53:32.193454 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:32.193415    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:32.193919 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:32.193688    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:32.194433 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:32.194415    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:34.944824 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:34.944787    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"]
Jul 22 02:53:34.945720 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:34.945659    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n" podUID="05894367-b97b-4f85-bf13-92d002462e1d" containerName="maas-api" containerID="cri-o://d40cde91378e76bd4ac6b80eba267491cdcb700256d5ef6144c81ff42ce082a3" gracePeriod=30
Jul 22 02:53:35.017751 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.017719    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:35.200242 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.200177    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"
Jul 22 02:53:35.269088 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.269046    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05894367-b97b-4f85-bf13-92d002462e1d-openshift-service-ca\") pod \"05894367-b97b-4f85-bf13-92d002462e1d\" (UID: \"05894367-b97b-4f85-bf13-92d002462e1d\") "
Jul 22 02:53:35.269252 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.269176    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05894367-b97b-4f85-bf13-92d002462e1d-maas-api-tls\") pod \"05894367-b97b-4f85-bf13-92d002462e1d\" (UID: \"05894367-b97b-4f85-bf13-92d002462e1d\") "
Jul 22 02:53:35.269252 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.269224    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8snp\" (UniqueName: \"kubernetes.io/projected/05894367-b97b-4f85-bf13-92d002462e1d-kube-api-access-v8snp\") pod \"05894367-b97b-4f85-bf13-92d002462e1d\" (UID: \"05894367-b97b-4f85-bf13-92d002462e1d\") "
Jul 22 02:53:35.269397 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.269374    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05894367-b97b-4f85-bf13-92d002462e1d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "05894367-b97b-4f85-bf13-92d002462e1d" (UID: "05894367-b97b-4f85-bf13-92d002462e1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:53:35.269470 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.269458    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05894367-b97b-4f85-bf13-92d002462e1d-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:53:35.271309 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.271279    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05894367-b97b-4f85-bf13-92d002462e1d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "05894367-b97b-4f85-bf13-92d002462e1d" (UID: "05894367-b97b-4f85-bf13-92d002462e1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:53:35.271419 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.271321    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05894367-b97b-4f85-bf13-92d002462e1d-kube-api-access-v8snp" (OuterVolumeSpecName: "kube-api-access-v8snp") pod "05894367-b97b-4f85-bf13-92d002462e1d" (UID: "05894367-b97b-4f85-bf13-92d002462e1d"). InnerVolumeSpecName "kube-api-access-v8snp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:53:35.369936 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.369899    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05894367-b97b-4f85-bf13-92d002462e1d-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:53:35.369936 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.369929    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8snp\" (UniqueName: \"kubernetes.io/projected/05894367-b97b-4f85-bf13-92d002462e1d-kube-api-access-v8snp\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:53:35.532733 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.532649    2565 generic.go:356] "Generic (PLEG): container finished" podID="05894367-b97b-4f85-bf13-92d002462e1d" containerID="d40cde91378e76bd4ac6b80eba267491cdcb700256d5ef6144c81ff42ce082a3" exitCode=0
Jul 22 02:53:35.532733 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.532692    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"
Jul 22 02:53:35.532733 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.532720    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n" event={"ID":"05894367-b97b-4f85-bf13-92d002462e1d","Type":"ContainerDied","Data":"d40cde91378e76bd4ac6b80eba267491cdcb700256d5ef6144c81ff42ce082a3"}
Jul 22 02:53:35.533072 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.532766    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n" event={"ID":"05894367-b97b-4f85-bf13-92d002462e1d","Type":"ContainerDied","Data":"011fd2fc17948c18b79573ff639648b768ebf05feed5ac8825014c1fc638cf92"}
Jul 22 02:53:35.533072 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.532790    2565 scope.go:117] "RemoveContainer" containerID="d40cde91378e76bd4ac6b80eba267491cdcb700256d5ef6144c81ff42ce082a3"
Jul 22 02:53:35.542425 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.542408    2565 scope.go:117] "RemoveContainer" containerID="d40cde91378e76bd4ac6b80eba267491cdcb700256d5ef6144c81ff42ce082a3"
Jul 22 02:53:35.542672 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:53:35.542653    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d40cde91378e76bd4ac6b80eba267491cdcb700256d5ef6144c81ff42ce082a3\": container with ID starting with d40cde91378e76bd4ac6b80eba267491cdcb700256d5ef6144c81ff42ce082a3 not found: ID does not exist" containerID="d40cde91378e76bd4ac6b80eba267491cdcb700256d5ef6144c81ff42ce082a3"
Jul 22 02:53:35.542727 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.542680    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d40cde91378e76bd4ac6b80eba267491cdcb700256d5ef6144c81ff42ce082a3"} err="failed to get container status \"d40cde91378e76bd4ac6b80eba267491cdcb700256d5ef6144c81ff42ce082a3\": rpc error: code = NotFound desc = could not find container \"d40cde91378e76bd4ac6b80eba267491cdcb700256d5ef6144c81ff42ce082a3\": container with ID starting with d40cde91378e76bd4ac6b80eba267491cdcb700256d5ef6144c81ff42ce082a3 not found: ID does not exist"
Jul 22 02:53:35.554156 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.554132    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"]
Jul 22 02:53:35.558524 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.558503    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0a5791b8-bf79545f8-qsx6n"]
Jul 22 02:53:35.726359 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:35.726326    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05894367-b97b-4f85-bf13-92d002462e1d" path="/var/lib/kubelet/pods/05894367-b97b-4f85-bf13-92d002462e1d/volumes"
Jul 22 02:53:39.453060 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:39.453033    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:39.837036 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:39.836952    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:45.721985 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:45.721910    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:45.821225 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:45.821198    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:46.019169 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:46.019081    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"]
Jul 22 02:53:46.019364 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:46.019340    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx" podUID="b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" containerName="istio-proxy" containerID="cri-o://e158b426b6a105e633a397a029aa4a4987104bd0534fe0a39aba3fc4596d4e8e" gracePeriod=30
Jul 22 02:53:46.033708 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:46.033681    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:46.225779 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:46.225746    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:46.360774 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:46.360735    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:46.453626 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:46.453595    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:46.573327 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:46.573299    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:47.196036 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:47.196004    2565 patch_prober.go:28] interesting pod/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 22 02:53:47.196394 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:47.196067    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx" podUID="b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 22 02:53:51.266674 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.266651    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:51.311630 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.311554    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istiod-ca-cert\") pod \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") "
Jul 22 02:53:51.311782 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.311641    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-workload-socket\") pod \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") "
Jul 22 02:53:51.311782 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.311669    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkrp7\" (UniqueName: \"kubernetes.io/projected/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-kube-api-access-jkrp7\") pod \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") "
Jul 22 02:53:51.311782 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.311705    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-podinfo\") pod \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") "
Jul 22 02:53:51.311782 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.311766    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-credential-socket\") pod \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") "
Jul 22 02:53:51.312067 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.311792    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-workload-certs\") pod \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") "
Jul 22 02:53:51.312067 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.311820    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-data\") pod \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") "
Jul 22 02:53:51.312067 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.311884    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-token\") pod \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") "
Jul 22 02:53:51.312067 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.311915    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-envoy\") pod \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\" (UID: \"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9\") "
Jul 22 02:53:51.312067 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.311964    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" (UID: "b4312d4e-2fb5-4c6a-ab25-39cb47c313c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:53:51.312067 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.311984    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" (UID: "b4312d4e-2fb5-4c6a-ab25-39cb47c313c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:53:51.312067 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.312033    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" (UID: "b4312d4e-2fb5-4c6a-ab25-39cb47c313c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:53:51.312470 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.312208    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" (UID: "b4312d4e-2fb5-4c6a-ab25-39cb47c313c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:53:51.312470 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.312317    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istiod-ca-cert\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:53:51.312470 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.312336    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-workload-socket\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:53:51.312470 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.312353    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-credential-socket\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:53:51.312470 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.312368    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-workload-certs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:53:51.312901 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.312881    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-data" (OuterVolumeSpecName: "istio-data") pod "b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" (UID: "b4312d4e-2fb5-4c6a-ab25-39cb47c313c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:53:51.313991 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.313963    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-kube-api-access-jkrp7" (OuterVolumeSpecName: "kube-api-access-jkrp7") pod "b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" (UID: "b4312d4e-2fb5-4c6a-ab25-39cb47c313c9"). InnerVolumeSpecName "kube-api-access-jkrp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:53:51.314269 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.314246    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" (UID: "b4312d4e-2fb5-4c6a-ab25-39cb47c313c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 22 02:53:51.314374 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.314357    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" (UID: "b4312d4e-2fb5-4c6a-ab25-39cb47c313c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:53:51.314459 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.314442    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-token" (OuterVolumeSpecName: "istio-token") pod "b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" (UID: "b4312d4e-2fb5-4c6a-ab25-39cb47c313c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:53:51.413419 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.413391    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-podinfo\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:53:51.413419 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.413415    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-data\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:53:51.413419 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.413424    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-token\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:53:51.413611 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.413432    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-istio-envoy\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:53:51.413611 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.413442    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkrp7\" (UniqueName: \"kubernetes.io/projected/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9-kube-api-access-jkrp7\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:53:51.617330 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.617296    2565 generic.go:356] "Generic (PLEG): container finished" podID="b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" containerID="e158b426b6a105e633a397a029aa4a4987104bd0534fe0a39aba3fc4596d4e8e" exitCode=0
Jul 22 02:53:51.617496 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.617344    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"
Jul 22 02:53:51.617496 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.617363    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx" event={"ID":"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9","Type":"ContainerDied","Data":"e158b426b6a105e633a397a029aa4a4987104bd0534fe0a39aba3fc4596d4e8e"}
Jul 22 02:53:51.617496 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.617406    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx" event={"ID":"b4312d4e-2fb5-4c6a-ab25-39cb47c313c9","Type":"ContainerDied","Data":"b85577cf56aad4733f2e7ceb345c03a16a366ed58325c4a841902c7c2885ba60"}
Jul 22 02:53:51.617496 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.617428    2565 scope.go:117] "RemoveContainer" containerID="e158b426b6a105e633a397a029aa4a4987104bd0534fe0a39aba3fc4596d4e8e"
Jul 22 02:53:51.626403 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.626383    2565 scope.go:117] "RemoveContainer" containerID="52530c100cdbfbd720da25c8e65a410c9c06efd64a2c166bc6d94835dac0450c"
Jul 22 02:53:51.633923 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.633905    2565 scope.go:117] "RemoveContainer" containerID="e158b426b6a105e633a397a029aa4a4987104bd0534fe0a39aba3fc4596d4e8e"
Jul 22 02:53:51.634171 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:53:51.634153    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e158b426b6a105e633a397a029aa4a4987104bd0534fe0a39aba3fc4596d4e8e\": container with ID starting with e158b426b6a105e633a397a029aa4a4987104bd0534fe0a39aba3fc4596d4e8e not found: ID does not exist" containerID="e158b426b6a105e633a397a029aa4a4987104bd0534fe0a39aba3fc4596d4e8e"
Jul 22 02:53:51.634236 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.634179    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e158b426b6a105e633a397a029aa4a4987104bd0534fe0a39aba3fc4596d4e8e"} err="failed to get container status \"e158b426b6a105e633a397a029aa4a4987104bd0534fe0a39aba3fc4596d4e8e\": rpc error: code = NotFound desc = could not find container \"e158b426b6a105e633a397a029aa4a4987104bd0534fe0a39aba3fc4596d4e8e\": container with ID starting with e158b426b6a105e633a397a029aa4a4987104bd0534fe0a39aba3fc4596d4e8e not found: ID does not exist"
Jul 22 02:53:51.634236 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.634195    2565 scope.go:117] "RemoveContainer" containerID="52530c100cdbfbd720da25c8e65a410c9c06efd64a2c166bc6d94835dac0450c"
Jul 22 02:53:51.634457 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:53:51.634440    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52530c100cdbfbd720da25c8e65a410c9c06efd64a2c166bc6d94835dac0450c\": container with ID starting with 52530c100cdbfbd720da25c8e65a410c9c06efd64a2c166bc6d94835dac0450c not found: ID does not exist" containerID="52530c100cdbfbd720da25c8e65a410c9c06efd64a2c166bc6d94835dac0450c"
Jul 22 02:53:51.634497 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.634462    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52530c100cdbfbd720da25c8e65a410c9c06efd64a2c166bc6d94835dac0450c"} err="failed to get container status \"52530c100cdbfbd720da25c8e65a410c9c06efd64a2c166bc6d94835dac0450c\": rpc error: code = NotFound desc = could not find container \"52530c100cdbfbd720da25c8e65a410c9c06efd64a2c166bc6d94835dac0450c\": container with ID starting with 52530c100cdbfbd720da25c8e65a410c9c06efd64a2c166bc6d94835dac0450c not found: ID does not exist"
Jul 22 02:53:51.639102 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.639082    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"]
Jul 22 02:53:51.642938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.642919    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0a5791b8-openshift-default-64785c86d7-8j5nx"]
Jul 22 02:53:51.726233 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:51.726193    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" path="/var/lib/kubelet/pods/b4312d4e-2fb5-4c6a-ab25-39cb47c313c9/volumes"
Jul 22 02:53:52.456237 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:52.456201    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:52.610438 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:52.610405    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:52.803562 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:52.803449    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:52.894115 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:52.894059    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:58.771654 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:58.771617    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:53:58.851238 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:53:58.851202    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:54:05.694423 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:05.694391    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:54:06.020029 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.019947    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"]
Jul 22 02:54:06.020306 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.020293    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" containerName="istio-proxy"
Jul 22 02:54:06.020370 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.020306    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" containerName="istio-proxy"
Jul 22 02:54:06.020370 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.020317    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05894367-b97b-4f85-bf13-92d002462e1d" containerName="maas-api"
Jul 22 02:54:06.020370 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.020324    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="05894367-b97b-4f85-bf13-92d002462e1d" containerName="maas-api"
Jul 22 02:54:06.020523 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.020384    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" containerName="istio-proxy"
Jul 22 02:54:06.020523 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.020395    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="05894367-b97b-4f85-bf13-92d002462e1d" containerName="maas-api"
Jul 22 02:54:06.023006 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.022990    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.025668 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.025644    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bd920c0c-openshift-default-dockercfg-td6xf\""
Jul 22 02:54:06.037215 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.037192    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"]
Jul 22 02:54:06.049703 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.049673    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:54:06.140002 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.139961    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895f9525-6d94-4801-9e0a-1897d0e38861-istio-token\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.140002 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.139996    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-credential-socket\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.140237 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.140060    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz4nq\" (UniqueName: \"kubernetes.io/projected/895f9525-6d94-4801-9e0a-1897d0e38861-kube-api-access-nz4nq\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.140237 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.140119    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-workload-certs\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.140237 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.140143    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-workload-socket\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.140237 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.140178    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895f9525-6d94-4801-9e0a-1897d0e38861-istio-podinfo\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.140237 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.140203    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-istio-data\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.140237 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.140231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-istio-envoy\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.140492 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.140315    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895f9525-6d94-4801-9e0a-1897d0e38861-istiod-ca-cert\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.240981 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.240951    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-workload-socket\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.241161 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.240991    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895f9525-6d94-4801-9e0a-1897d0e38861-istio-podinfo\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.241161 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.241012    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-istio-data\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.241280 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.241166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-istio-envoy\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.241280 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.241236    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895f9525-6d94-4801-9e0a-1897d0e38861-istiod-ca-cert\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.241391 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.241293    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895f9525-6d94-4801-9e0a-1897d0e38861-istio-token\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.241391 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.241332    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-credential-socket\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.241493 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.241387    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz4nq\" (UniqueName: \"kubernetes.io/projected/895f9525-6d94-4801-9e0a-1897d0e38861-kube-api-access-nz4nq\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.241493 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.241389    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-workload-socket\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.241493 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.241425    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-workload-certs\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.241493 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.241426    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-istio-data\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.241683 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.241643    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-credential-socket\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.241756 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.241734    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-workload-certs\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.241970 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.241950    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895f9525-6d94-4801-9e0a-1897d0e38861-istiod-ca-cert\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.243742 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.243722    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895f9525-6d94-4801-9e0a-1897d0e38861-istio-podinfo\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.243864 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.243762    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-istio-envoy\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.250962 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.250939    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895f9525-6d94-4801-9e0a-1897d0e38861-istio-token\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.251602 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.251582    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz4nq\" (UniqueName: \"kubernetes.io/projected/895f9525-6d94-4801-9e0a-1897d0e38861-kube-api-access-nz4nq\") pod \"e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") " pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.333504 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.333434    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:06.510511 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.510478    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"]
Jul 22 02:54:06.515657 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:54:06.515628    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895f9525_6d94_4801_9e0a_1897d0e38861.slice/crio-d91ddaf3c22ea93b3bb37e8586ff6f6691863a7e854ac73729d9242f60b861b1 WatchSource:0}: Error finding container d91ddaf3c22ea93b3bb37e8586ff6f6691863a7e854ac73729d9242f60b861b1: Status 404 returned error can't find the container with id d91ddaf3c22ea93b3bb37e8586ff6f6691863a7e854ac73729d9242f60b861b1
Jul 22 02:54:06.517862 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.517820    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:54:06.517939 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.517906    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:54:06.517979 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.517937    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:54:06.687914 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.687876    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx" event={"ID":"895f9525-6d94-4801-9e0a-1897d0e38861","Type":"ContainerStarted","Data":"1c59f713f8ada509674737345b7894e72f73a2f2e54aa4f27cc42eed26e58db3"}
Jul 22 02:54:06.688115 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.687949    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx" event={"ID":"895f9525-6d94-4801-9e0a-1897d0e38861","Type":"ContainerStarted","Data":"d91ddaf3c22ea93b3bb37e8586ff6f6691863a7e854ac73729d9242f60b861b1"}
Jul 22 02:54:06.711445 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.711376    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx" podStartSLOduration=1.711355105 podStartE2EDuration="1.711355105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:54:06.710111873 +0000 UTC m=+1997.558448448" watchObservedRunningTime="2026-07-22 02:54:06.711355105 +0000 UTC m=+1997.559691680"
Jul 22 02:54:06.938481 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:06.938397    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:54:07.334541 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:07.334467    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:07.339181 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:07.339159    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:07.692823 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:07.692792    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:07.693618 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:07.693588    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:08.676578 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:08.676543    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:54:08.768649 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:08.768617    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"]
Jul 22 02:54:08.768985 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:08.768971    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" containerName="istio-proxy"
Jul 22 02:54:08.769037 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:08.768986    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" containerName="istio-proxy"
Jul 22 02:54:08.769070 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:08.769060    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4312d4e-2fb5-4c6a-ab25-39cb47c313c9" containerName="istio-proxy"
Jul 22 02:54:08.770972 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:08.770957    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"
Jul 22 02:54:08.773674 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:08.773656    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-bd920c0c-serving-cert\""
Jul 22 02:54:08.786562 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:08.786540    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"]
Jul 22 02:54:08.796388 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:08.796345    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:54:08.868281 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:08.868247    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a6d693-99aa-4e57-9fbd-f8058283c758-openshift-service-ca\") pod \"maas-api-e2e-mt-bd920c0c-69949d4468-zm266\" (UID: \"64a6d693-99aa-4e57-9fbd-f8058283c758\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"
Jul 22 02:54:08.868491 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:08.868295    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hbnf\" (UniqueName: \"kubernetes.io/projected/64a6d693-99aa-4e57-9fbd-f8058283c758-kube-api-access-9hbnf\") pod \"maas-api-e2e-mt-bd920c0c-69949d4468-zm266\" (UID: \"64a6d693-99aa-4e57-9fbd-f8058283c758\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"
Jul 22 02:54:08.868491 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:08.868354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64a6d693-99aa-4e57-9fbd-f8058283c758-maas-api-tls\") pod \"maas-api-e2e-mt-bd920c0c-69949d4468-zm266\" (UID: \"64a6d693-99aa-4e57-9fbd-f8058283c758\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"
Jul 22 02:54:08.890587 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:08.890556    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:54:08.968919 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:08.968821    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64a6d693-99aa-4e57-9fbd-f8058283c758-maas-api-tls\") pod \"maas-api-e2e-mt-bd920c0c-69949d4468-zm266\" (UID: \"64a6d693-99aa-4e57-9fbd-f8058283c758\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"
Jul 22 02:54:08.969071 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:08.968944    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a6d693-99aa-4e57-9fbd-f8058283c758-openshift-service-ca\") pod \"maas-api-e2e-mt-bd920c0c-69949d4468-zm266\" (UID: \"64a6d693-99aa-4e57-9fbd-f8058283c758\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"
Jul 22 02:54:08.969071 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:54:08.968966    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-serving-cert: secret "maas-api-e2e-mt-bd920c0c-serving-cert" not found
Jul 22 02:54:08.969071 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:08.968970    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hbnf\" (UniqueName: \"kubernetes.io/projected/64a6d693-99aa-4e57-9fbd-f8058283c758-kube-api-access-9hbnf\") pod \"maas-api-e2e-mt-bd920c0c-69949d4468-zm266\" (UID: \"64a6d693-99aa-4e57-9fbd-f8058283c758\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"
Jul 22 02:54:08.969071 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:54:08.969034    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a6d693-99aa-4e57-9fbd-f8058283c758-maas-api-tls podName:64a6d693-99aa-4e57-9fbd-f8058283c758 nodeName:}" failed. No retries permitted until 2026-07-22 02:54:09.46901303 +0000 UTC m=+2000.317349599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/64a6d693-99aa-4e57-9fbd-f8058283c758-maas-api-tls") pod "maas-api-e2e-mt-bd920c0c-69949d4468-zm266" (UID: "64a6d693-99aa-4e57-9fbd-f8058283c758") : secret "maas-api-e2e-mt-bd920c0c-serving-cert" not found
Jul 22 02:54:08.969565 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:08.969547    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a6d693-99aa-4e57-9fbd-f8058283c758-openshift-service-ca\") pod \"maas-api-e2e-mt-bd920c0c-69949d4468-zm266\" (UID: \"64a6d693-99aa-4e57-9fbd-f8058283c758\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"
Jul 22 02:54:08.982666 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:08.982643    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hbnf\" (UniqueName: \"kubernetes.io/projected/64a6d693-99aa-4e57-9fbd-f8058283c758-kube-api-access-9hbnf\") pod \"maas-api-e2e-mt-bd920c0c-69949d4468-zm266\" (UID: \"64a6d693-99aa-4e57-9fbd-f8058283c758\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"
Jul 22 02:54:09.106970 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:09.106936    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:54:09.427157 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:09.427123    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:54:09.474058 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:09.474027    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64a6d693-99aa-4e57-9fbd-f8058283c758-maas-api-tls\") pod \"maas-api-e2e-mt-bd920c0c-69949d4468-zm266\" (UID: \"64a6d693-99aa-4e57-9fbd-f8058283c758\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"
Jul 22 02:54:09.476481 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:09.476456    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64a6d693-99aa-4e57-9fbd-f8058283c758-maas-api-tls\") pod \"maas-api-e2e-mt-bd920c0c-69949d4468-zm266\" (UID: \"64a6d693-99aa-4e57-9fbd-f8058283c758\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"
Jul 22 02:54:09.527671 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:09.527626    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:54:09.681602 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:09.681496    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"
Jul 22 02:54:09.822110 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:09.822085    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"]
Jul 22 02:54:09.824117 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:54:09.824089    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a6d693_99aa_4e57_9fbd_f8058283c758.slice/crio-0eedc4c160cc60c99b851663a2499c72900e2e5544b9a5d109ce5070f8afa03a WatchSource:0}: Error finding container 0eedc4c160cc60c99b851663a2499c72900e2e5544b9a5d109ce5070f8afa03a: Status 404 returned error can't find the container with id 0eedc4c160cc60c99b851663a2499c72900e2e5544b9a5d109ce5070f8afa03a
Jul 22 02:54:09.825536 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:09.825516    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 02:54:10.711470 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:10.711436    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266" event={"ID":"64a6d693-99aa-4e57-9fbd-f8058283c758","Type":"ContainerStarted","Data":"a267ec5df54fb37336a199f607a3f63fbd906d92018abcf8b990cfac9bf2d150"}
Jul 22 02:54:10.711882 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:10.711498    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266" event={"ID":"64a6d693-99aa-4e57-9fbd-f8058283c758","Type":"ContainerStarted","Data":"0eedc4c160cc60c99b851663a2499c72900e2e5544b9a5d109ce5070f8afa03a"}
Jul 22 02:54:10.711882 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:10.711525    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"
Jul 22 02:54:10.728959 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:10.728910    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266" podStartSLOduration=2.436613489 podStartE2EDuration="2.728894778s" totalImagesPullingTime="292.281289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:54:10.728242092 +0000 UTC m=+2001.576578667" watchObservedRunningTime="2026-07-22 02:54:10.728894778 +0000 UTC m=+2001.577231353"
Jul 22 02:54:13.165484 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:13.165445    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:54:16.726983 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:16.726943    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"
Jul 22 02:54:23.136361 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:23.136327    2565 patch_prober.go:28] interesting pod/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": read tcp 10.133.0.2:54432->10.133.0.53:15021: read: connection reset by peer" start-of-body=
Jul 22 02:54:23.136759 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:23.136385    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx" podUID="895f9525-6d94-4801-9e0a-1897d0e38861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": read tcp 10.133.0.2:54432->10.133.0.53:15021: read: connection reset by peer"
Jul 22 02:54:23.775090 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:23.775056    2565 generic.go:356] "Generic (PLEG): container finished" podID="895f9525-6d94-4801-9e0a-1897d0e38861" containerID="1c59f713f8ada509674737345b7894e72f73a2f2e54aa4f27cc42eed26e58db3" exitCode=0
Jul 22 02:54:23.775262 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:23.775124    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx" event={"ID":"895f9525-6d94-4801-9e0a-1897d0e38861","Type":"ContainerDied","Data":"1c59f713f8ada509674737345b7894e72f73a2f2e54aa4f27cc42eed26e58db3"}
Jul 22 02:54:23.775529 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:23.775514    2565 scope.go:117] "RemoveContainer" containerID="1c59f713f8ada509674737345b7894e72f73a2f2e54aa4f27cc42eed26e58db3"
Jul 22 02:54:23.776515 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:23.776488    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:54:23.776570 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:23.776553    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:54:23.776606 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:23.776585    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:54:24.781280 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:24.781246    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx" event={"ID":"895f9525-6d94-4801-9e0a-1897d0e38861","Type":"ContainerStarted","Data":"adc39b46ea18d9526641cd25782340178787a8a4541835a41442e601a213d729"}
Jul 22 02:54:25.333959 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:25.333920    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:25.334576 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:25.334278    2565 patch_prober.go:28] interesting pod/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 22 02:54:25.334770 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:25.334745    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx" podUID="895f9525-6d94-4801-9e0a-1897d0e38861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 22 02:54:26.043116 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:26.043074    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:54:26.334342 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:26.334250    2565 patch_prober.go:28] interesting pod/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 22 02:54:26.334603 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:26.334571    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx" podUID="895f9525-6d94-4801-9e0a-1897d0e38861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 22 02:54:27.334030 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:27.333985    2565 patch_prober.go:28] interesting pod/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 22 02:54:27.334527 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:27.334055    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx" podUID="895f9525-6d94-4801-9e0a-1897d0e38861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 22 02:54:28.286339 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.286307    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"]
Jul 22 02:54:28.288589 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.288573    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:28.291015 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.290994    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-bd920c0c\"/\"default-dockercfg-cpdqf\""
Jul 22 02:54:28.291392 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.291376    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-bd920c0c\"/\"kube-root-ca.crt\""
Jul 22 02:54:28.292059 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.292042    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-bd920c0c\"/\"e2e-lifecycle-model-bd920c0c-kserve-self-signed-certs\""
Jul 22 02:54:28.292127 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.292064    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-bd920c0c\"/\"openshift-service-ca.crt\""
Jul 22 02:54:28.305340 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.305314    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"]
Jul 22 02:54:28.324920 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.324892    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:54:28.336822 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.336791    2565 patch_prober.go:28] interesting pod/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 22 02:54:28.337362 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.337333    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx" podUID="895f9525-6d94-4801-9e0a-1897d0e38861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 22 02:54:28.447395 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.447362    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-model-cache\") pod \"e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") " pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:28.447395 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.447409    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-dshm\") pod \"e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") " pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:28.447657 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.447441    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdfgk\" (UniqueName: \"kubernetes.io/projected/6f84e6d5-c430-4378-b86b-901c90231a84-kube-api-access-gdfgk\") pod \"e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") " pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:28.447657 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.447487    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f84e6d5-c430-4378-b86b-901c90231a84-tls-certs\") pod \"e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") " pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:28.447657 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.447503    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-home\") pod \"e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") " pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:28.447657 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.447555    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-tmp-dir\") pod \"e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") " pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:28.548517 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.548426    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f84e6d5-c430-4378-b86b-901c90231a84-tls-certs\") pod \"e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") " pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:28.548517 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.548465    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-home\") pod \"e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") " pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:28.548517 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.548482    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-tmp-dir\") pod \"e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") " pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:28.548777 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.548525    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-model-cache\") pod \"e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") " pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:28.548777 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.548557    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-dshm\") pod \"e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") " pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:28.548777 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.548583    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdfgk\" (UniqueName: \"kubernetes.io/projected/6f84e6d5-c430-4378-b86b-901c90231a84-kube-api-access-gdfgk\") pod \"e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") " pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:28.549053 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.548984    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-model-cache\") pod \"e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") " pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:28.549053 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.549011    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-home\") pod \"e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") " pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:28.549147 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.549070    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-tmp-dir\") pod \"e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") " pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:28.550758 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.550731    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-dshm\") pod \"e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") " pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:28.551064 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.551032    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f84e6d5-c430-4378-b86b-901c90231a84-tls-certs\") pod \"e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") " pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:28.559664 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.559641    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdfgk\" (UniqueName: \"kubernetes.io/projected/6f84e6d5-c430-4378-b86b-901c90231a84-kube-api-access-gdfgk\") pod \"e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") " pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:28.597431 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.597403    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:28.781064 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.781038    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"]
Jul 22 02:54:28.783262 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:54:28.783239    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f84e6d5_c430_4378_b86b_901c90231a84.slice/crio-89d91c0abf6b5323eb0f5a40e9a7c11a6b4174ee26b2d4ad87a122e367e67471 WatchSource:0}: Error finding container 89d91c0abf6b5323eb0f5a40e9a7c11a6b4174ee26b2d4ad87a122e367e67471: Status 404 returned error can't find the container with id 89d91c0abf6b5323eb0f5a40e9a7c11a6b4174ee26b2d4ad87a122e367e67471
Jul 22 02:54:28.804215 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:28.804160    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs" event={"ID":"6f84e6d5-c430-4378-b86b-901c90231a84","Type":"ContainerStarted","Data":"89d91c0abf6b5323eb0f5a40e9a7c11a6b4174ee26b2d4ad87a122e367e67471"}
Jul 22 02:54:29.338706 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:29.338670    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:29.339165 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:29.339047    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:29.346347 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:29.346321    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:54:29.810276 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:29.810237    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs" event={"ID":"6f84e6d5-c430-4378-b86b-901c90231a84","Type":"ContainerStarted","Data":"04552208166a9162712a33ee6bd2b2e2bcc07179fc8d23a6d79db49d2a190efb"}
Jul 22 02:54:29.834680 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:29.834622    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs" podStartSLOduration=1.834605263 podStartE2EDuration="1.834605263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:54:29.833670978 +0000 UTC m=+2020.682007552" watchObservedRunningTime="2026-07-22 02:54:29.834605263 +0000 UTC m=+2020.682941838"
Jul 22 02:54:38.597742 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:38.597703    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:38.598266 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:38.597777    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:38.610358 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:38.610333    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:54:38.864066 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:54:38.864041    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:55:02.137750 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.137717    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"]
Jul 22 02:55:02.138338 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.138006    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266" podUID="64a6d693-99aa-4e57-9fbd-f8058283c758" containerName="maas-api" containerID="cri-o://a267ec5df54fb37336a199f607a3f63fbd906d92018abcf8b990cfac9bf2d150" gracePeriod=30
Jul 22 02:55:02.208214 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.208180    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:55:02.384518 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.384496    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"
Jul 22 02:55:02.437561 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.437481    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a6d693-99aa-4e57-9fbd-f8058283c758-openshift-service-ca\") pod \"64a6d693-99aa-4e57-9fbd-f8058283c758\" (UID: \"64a6d693-99aa-4e57-9fbd-f8058283c758\") "
Jul 22 02:55:02.437561 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.437528    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hbnf\" (UniqueName: \"kubernetes.io/projected/64a6d693-99aa-4e57-9fbd-f8058283c758-kube-api-access-9hbnf\") pod \"64a6d693-99aa-4e57-9fbd-f8058283c758\" (UID: \"64a6d693-99aa-4e57-9fbd-f8058283c758\") "
Jul 22 02:55:02.437561 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.437560    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64a6d693-99aa-4e57-9fbd-f8058283c758-maas-api-tls\") pod \"64a6d693-99aa-4e57-9fbd-f8058283c758\" (UID: \"64a6d693-99aa-4e57-9fbd-f8058283c758\") "
Jul 22 02:55:02.437893 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.437865    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a6d693-99aa-4e57-9fbd-f8058283c758-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "64a6d693-99aa-4e57-9fbd-f8058283c758" (UID: "64a6d693-99aa-4e57-9fbd-f8058283c758"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:55:02.439594 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.439567    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a6d693-99aa-4e57-9fbd-f8058283c758-kube-api-access-9hbnf" (OuterVolumeSpecName: "kube-api-access-9hbnf") pod "64a6d693-99aa-4e57-9fbd-f8058283c758" (UID: "64a6d693-99aa-4e57-9fbd-f8058283c758"). InnerVolumeSpecName "kube-api-access-9hbnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:55:02.439729 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.439631    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64a6d693-99aa-4e57-9fbd-f8058283c758-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "64a6d693-99aa-4e57-9fbd-f8058283c758" (UID: "64a6d693-99aa-4e57-9fbd-f8058283c758"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:55:02.538495 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.538464    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a6d693-99aa-4e57-9fbd-f8058283c758-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:55:02.538495 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.538491    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hbnf\" (UniqueName: \"kubernetes.io/projected/64a6d693-99aa-4e57-9fbd-f8058283c758-kube-api-access-9hbnf\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:55:02.538495 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.538502    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64a6d693-99aa-4e57-9fbd-f8058283c758-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:55:02.968007 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.967976    2565 generic.go:356] "Generic (PLEG): container finished" podID="64a6d693-99aa-4e57-9fbd-f8058283c758" containerID="a267ec5df54fb37336a199f607a3f63fbd906d92018abcf8b990cfac9bf2d150" exitCode=0
Jul 22 02:55:02.968178 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.968038    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"
Jul 22 02:55:02.968178 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.968060    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266" event={"ID":"64a6d693-99aa-4e57-9fbd-f8058283c758","Type":"ContainerDied","Data":"a267ec5df54fb37336a199f607a3f63fbd906d92018abcf8b990cfac9bf2d150"}
Jul 22 02:55:02.968178 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.968119    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266" event={"ID":"64a6d693-99aa-4e57-9fbd-f8058283c758","Type":"ContainerDied","Data":"0eedc4c160cc60c99b851663a2499c72900e2e5544b9a5d109ce5070f8afa03a"}
Jul 22 02:55:02.968178 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.968141    2565 scope.go:117] "RemoveContainer" containerID="a267ec5df54fb37336a199f607a3f63fbd906d92018abcf8b990cfac9bf2d150"
Jul 22 02:55:02.977156 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.977140    2565 scope.go:117] "RemoveContainer" containerID="a267ec5df54fb37336a199f607a3f63fbd906d92018abcf8b990cfac9bf2d150"
Jul 22 02:55:02.977379 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:55:02.977362    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a267ec5df54fb37336a199f607a3f63fbd906d92018abcf8b990cfac9bf2d150\": container with ID starting with a267ec5df54fb37336a199f607a3f63fbd906d92018abcf8b990cfac9bf2d150 not found: ID does not exist" containerID="a267ec5df54fb37336a199f607a3f63fbd906d92018abcf8b990cfac9bf2d150"
Jul 22 02:55:02.977463 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.977387    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a267ec5df54fb37336a199f607a3f63fbd906d92018abcf8b990cfac9bf2d150"} err="failed to get container status \"a267ec5df54fb37336a199f607a3f63fbd906d92018abcf8b990cfac9bf2d150\": rpc error: code = NotFound desc = could not find container \"a267ec5df54fb37336a199f607a3f63fbd906d92018abcf8b990cfac9bf2d150\": container with ID starting with a267ec5df54fb37336a199f607a3f63fbd906d92018abcf8b990cfac9bf2d150 not found: ID does not exist"
Jul 22 02:55:02.994440 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.994419    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"]
Jul 22 02:55:03.000025 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:02.999999    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd920c0c-69949d4468-zm266"]
Jul 22 02:55:03.725763 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:03.725727    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64a6d693-99aa-4e57-9fbd-f8058283c758" path="/var/lib/kubelet/pods/64a6d693-99aa-4e57-9fbd-f8058283c758/volumes"
Jul 22 02:55:06.768482 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:06.768451    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:55:09.108197 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:09.108164    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:55:14.068545 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:14.068466    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"]
Jul 22 02:55:14.144141 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:14.144111    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:55:14.220920 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:14.220886    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:55:29.088334 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:29.088271    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs" podUID="6f84e6d5-c430-4378-b86b-901c90231a84" containerName="main" containerID="cri-o://04552208166a9162712a33ee6bd2b2e2bcc07179fc8d23a6d79db49d2a190efb" gracePeriod=45
Jul 22 02:55:29.344040 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:29.343984    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:55:29.365418 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:29.365392    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-home\") pod \"6f84e6d5-c430-4378-b86b-901c90231a84\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") "
Jul 22 02:55:29.365542 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:29.365424    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdfgk\" (UniqueName: \"kubernetes.io/projected/6f84e6d5-c430-4378-b86b-901c90231a84-kube-api-access-gdfgk\") pod \"6f84e6d5-c430-4378-b86b-901c90231a84\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") "
Jul 22 02:55:29.365542 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:29.365453    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-tmp-dir\") pod \"6f84e6d5-c430-4378-b86b-901c90231a84\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") "
Jul 22 02:55:29.365542 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:29.365485    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f84e6d5-c430-4378-b86b-901c90231a84-tls-certs\") pod \"6f84e6d5-c430-4378-b86b-901c90231a84\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") "
Jul 22 02:55:29.365542 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:29.365501    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-dshm\") pod \"6f84e6d5-c430-4378-b86b-901c90231a84\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") "
Jul 22 02:55:29.365542 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:29.365539    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-model-cache\") pod \"6f84e6d5-c430-4378-b86b-901c90231a84\" (UID: \"6f84e6d5-c430-4378-b86b-901c90231a84\") "
Jul 22 02:55:29.365792 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:29.365705    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-home" (OuterVolumeSpecName: "home") pod "6f84e6d5-c430-4378-b86b-901c90231a84" (UID: "6f84e6d5-c430-4378-b86b-901c90231a84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:55:29.365792 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:29.365762    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f84e6d5-c430-4378-b86b-901c90231a84" (UID: "6f84e6d5-c430-4378-b86b-901c90231a84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:55:29.365934 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:29.365875    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-home\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:55:29.365934 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:29.365896    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-tmp-dir\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:55:29.366038 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:29.365964    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f84e6d5-c430-4378-b86b-901c90231a84" (UID: "6f84e6d5-c430-4378-b86b-901c90231a84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:55:29.368326 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:29.368140    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f84e6d5-c430-4378-b86b-901c90231a84-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f84e6d5-c430-4378-b86b-901c90231a84" (UID: "6f84e6d5-c430-4378-b86b-901c90231a84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:55:29.368326 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:29.368161    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-dshm" (OuterVolumeSpecName: "dshm") pod "6f84e6d5-c430-4378-b86b-901c90231a84" (UID: "6f84e6d5-c430-4378-b86b-901c90231a84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:55:29.368326 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:29.368222    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f84e6d5-c430-4378-b86b-901c90231a84-kube-api-access-gdfgk" (OuterVolumeSpecName: "kube-api-access-gdfgk") pod "6f84e6d5-c430-4378-b86b-901c90231a84" (UID: "6f84e6d5-c430-4378-b86b-901c90231a84"). InnerVolumeSpecName "kube-api-access-gdfgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:55:29.466358 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:29.466315    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-model-cache\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:55:29.466358 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:29.466354    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdfgk\" (UniqueName: \"kubernetes.io/projected/6f84e6d5-c430-4378-b86b-901c90231a84-kube-api-access-gdfgk\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:55:29.466358 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:29.466365    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f84e6d5-c430-4378-b86b-901c90231a84-tls-certs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:55:29.466358 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:29.466373    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f84e6d5-c430-4378-b86b-901c90231a84-dshm\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:55:30.102562 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:30.102533    2565 generic.go:356] "Generic (PLEG): container finished" podID="6f84e6d5-c430-4378-b86b-901c90231a84" containerID="04552208166a9162712a33ee6bd2b2e2bcc07179fc8d23a6d79db49d2a190efb" exitCode=0
Jul 22 02:55:30.102978 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:30.102575    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"
Jul 22 02:55:30.102978 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:30.102578    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs" event={"ID":"6f84e6d5-c430-4378-b86b-901c90231a84","Type":"ContainerDied","Data":"04552208166a9162712a33ee6bd2b2e2bcc07179fc8d23a6d79db49d2a190efb"}
Jul 22 02:55:30.102978 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:30.102699    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs" event={"ID":"6f84e6d5-c430-4378-b86b-901c90231a84","Type":"ContainerDied","Data":"89d91c0abf6b5323eb0f5a40e9a7c11a6b4174ee26b2d4ad87a122e367e67471"}
Jul 22 02:55:30.102978 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:30.102722    2565 scope.go:117] "RemoveContainer" containerID="04552208166a9162712a33ee6bd2b2e2bcc07179fc8d23a6d79db49d2a190efb"
Jul 22 02:55:30.111312 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:30.111298    2565 scope.go:117] "RemoveContainer" containerID="04552208166a9162712a33ee6bd2b2e2bcc07179fc8d23a6d79db49d2a190efb"
Jul 22 02:55:30.111559 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:55:30.111538    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04552208166a9162712a33ee6bd2b2e2bcc07179fc8d23a6d79db49d2a190efb\": container with ID starting with 04552208166a9162712a33ee6bd2b2e2bcc07179fc8d23a6d79db49d2a190efb not found: ID does not exist" containerID="04552208166a9162712a33ee6bd2b2e2bcc07179fc8d23a6d79db49d2a190efb"
Jul 22 02:55:30.111655 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:30.111563    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04552208166a9162712a33ee6bd2b2e2bcc07179fc8d23a6d79db49d2a190efb"} err="failed to get container status \"04552208166a9162712a33ee6bd2b2e2bcc07179fc8d23a6d79db49d2a190efb\": rpc error: code = NotFound desc = could not find container \"04552208166a9162712a33ee6bd2b2e2bcc07179fc8d23a6d79db49d2a190efb\": container with ID starting with 04552208166a9162712a33ee6bd2b2e2bcc07179fc8d23a6d79db49d2a190efb not found: ID does not exist"
Jul 22 02:55:30.119653 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:30.119628    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"]
Jul 22 02:55:30.124683 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:30.124663    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-bd920c0c/e2e-lifecycle-model-bd920c0c-kserve-5bf996f8f9-zfbbs"]
Jul 22 02:55:31.726481 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:31.726448    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f84e6d5-c430-4378-b86b-901c90231a84" path="/var/lib/kubelet/pods/6f84e6d5-c430-4378-b86b-901c90231a84/volumes"
Jul 22 02:55:35.784225 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:35.784189    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:55:35.884490 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:35.884460    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:55:51.320286 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:51.320245    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:55:52.349234 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:52.349203    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:55:58.941152 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:55:58.941116    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:56:04.496939 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:04.496905    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:56:04.598531 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:04.598498    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:56:04.761773 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:04.761693    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"]
Jul 22 02:56:04.761952 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:04.761927    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx" podUID="895f9525-6d94-4801-9e0a-1897d0e38861" containerName="istio-proxy" containerID="cri-o://adc39b46ea18d9526641cd25782340178787a8a4541835a41442e601a213d729" gracePeriod=30
Jul 22 02:56:04.794447 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:04.794417    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:56:05.114550 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:05.114519    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:56:06.167457 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:06.167419    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:56:06.265631 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:06.265603    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:56:06.365989 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:06.365952    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:56:07.096393 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:07.096352    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:56:07.198894 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:07.198843    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:56:07.298926 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:07.298893    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:56:08.692458 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:08.692421    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:56:10.025938 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.025876    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:56:10.096321 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.096288    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-credential-socket\") pod \"895f9525-6d94-4801-9e0a-1897d0e38861\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") "
Jul 22 02:56:10.096321 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.096327    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-workload-socket\") pod \"895f9525-6d94-4801-9e0a-1897d0e38861\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") "
Jul 22 02:56:10.096574 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.096357    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895f9525-6d94-4801-9e0a-1897d0e38861-istio-token\") pod \"895f9525-6d94-4801-9e0a-1897d0e38861\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") "
Jul 22 02:56:10.096574 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.096374    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895f9525-6d94-4801-9e0a-1897d0e38861-istio-podinfo\") pod \"895f9525-6d94-4801-9e0a-1897d0e38861\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") "
Jul 22 02:56:10.096574 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.096396    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nz4nq\" (UniqueName: \"kubernetes.io/projected/895f9525-6d94-4801-9e0a-1897d0e38861-kube-api-access-nz4nq\") pod \"895f9525-6d94-4801-9e0a-1897d0e38861\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") "
Jul 22 02:56:10.096574 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.096418    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-istio-envoy\") pod \"895f9525-6d94-4801-9e0a-1897d0e38861\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") "
Jul 22 02:56:10.096574 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.096481    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-workload-certs\") pod \"895f9525-6d94-4801-9e0a-1897d0e38861\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") "
Jul 22 02:56:10.096574 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.096551    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-istio-data\") pod \"895f9525-6d94-4801-9e0a-1897d0e38861\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") "
Jul 22 02:56:10.096914 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.096599    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895f9525-6d94-4801-9e0a-1897d0e38861-istiod-ca-cert\") pod \"895f9525-6d94-4801-9e0a-1897d0e38861\" (UID: \"895f9525-6d94-4801-9e0a-1897d0e38861\") "
Jul 22 02:56:10.096914 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.096597    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "895f9525-6d94-4801-9e0a-1897d0e38861" (UID: "895f9525-6d94-4801-9e0a-1897d0e38861"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:56:10.096914 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.096705    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "895f9525-6d94-4801-9e0a-1897d0e38861" (UID: "895f9525-6d94-4801-9e0a-1897d0e38861"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:56:10.097073 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.096898    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "895f9525-6d94-4801-9e0a-1897d0e38861" (UID: "895f9525-6d94-4801-9e0a-1897d0e38861"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:56:10.097073 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.096950    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-credential-socket\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:56:10.097073 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.096973    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-workload-socket\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:56:10.097073 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.097034    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895f9525-6d94-4801-9e0a-1897d0e38861-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "895f9525-6d94-4801-9e0a-1897d0e38861" (UID: "895f9525-6d94-4801-9e0a-1897d0e38861"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:56:10.097802 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.097775    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-istio-data" (OuterVolumeSpecName: "istio-data") pod "895f9525-6d94-4801-9e0a-1897d0e38861" (UID: "895f9525-6d94-4801-9e0a-1897d0e38861"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:56:10.098828 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.098806    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895f9525-6d94-4801-9e0a-1897d0e38861-istio-token" (OuterVolumeSpecName: "istio-token") pod "895f9525-6d94-4801-9e0a-1897d0e38861" (UID: "895f9525-6d94-4801-9e0a-1897d0e38861"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:56:10.098972 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.098839    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/895f9525-6d94-4801-9e0a-1897d0e38861-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "895f9525-6d94-4801-9e0a-1897d0e38861" (UID: "895f9525-6d94-4801-9e0a-1897d0e38861"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 22 02:56:10.098972 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.098932    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895f9525-6d94-4801-9e0a-1897d0e38861-kube-api-access-nz4nq" (OuterVolumeSpecName: "kube-api-access-nz4nq") pod "895f9525-6d94-4801-9e0a-1897d0e38861" (UID: "895f9525-6d94-4801-9e0a-1897d0e38861"). InnerVolumeSpecName "kube-api-access-nz4nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:56:10.099261 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.099241    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "895f9525-6d94-4801-9e0a-1897d0e38861" (UID: "895f9525-6d94-4801-9e0a-1897d0e38861"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:56:10.198128 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.198097    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-workload-certs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:56:10.198128 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.198121    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-istio-data\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:56:10.198128 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.198129    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895f9525-6d94-4801-9e0a-1897d0e38861-istiod-ca-cert\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:56:10.198364 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.198137    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895f9525-6d94-4801-9e0a-1897d0e38861-istio-token\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:56:10.198364 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.198146    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895f9525-6d94-4801-9e0a-1897d0e38861-istio-podinfo\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:56:10.198364 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.198154    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz4nq\" (UniqueName: \"kubernetes.io/projected/895f9525-6d94-4801-9e0a-1897d0e38861-kube-api-access-nz4nq\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:56:10.198364 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.198163    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895f9525-6d94-4801-9e0a-1897d0e38861-istio-envoy\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:56:10.295634 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.295539    2565 generic.go:356] "Generic (PLEG): container finished" podID="895f9525-6d94-4801-9e0a-1897d0e38861" containerID="adc39b46ea18d9526641cd25782340178787a8a4541835a41442e601a213d729" exitCode=0
Jul 22 02:56:10.295634 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.295597    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"
Jul 22 02:56:10.295634 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.295622    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx" event={"ID":"895f9525-6d94-4801-9e0a-1897d0e38861","Type":"ContainerDied","Data":"adc39b46ea18d9526641cd25782340178787a8a4541835a41442e601a213d729"}
Jul 22 02:56:10.295928 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.295678    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx" event={"ID":"895f9525-6d94-4801-9e0a-1897d0e38861","Type":"ContainerDied","Data":"d91ddaf3c22ea93b3bb37e8586ff6f6691863a7e854ac73729d9242f60b861b1"}
Jul 22 02:56:10.295928 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.295702    2565 scope.go:117] "RemoveContainer" containerID="adc39b46ea18d9526641cd25782340178787a8a4541835a41442e601a213d729"
Jul 22 02:56:10.305276 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.305257    2565 scope.go:117] "RemoveContainer" containerID="1c59f713f8ada509674737345b7894e72f73a2f2e54aa4f27cc42eed26e58db3"
Jul 22 02:56:10.313452 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.313434    2565 scope.go:117] "RemoveContainer" containerID="adc39b46ea18d9526641cd25782340178787a8a4541835a41442e601a213d729"
Jul 22 02:56:10.313707 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:56:10.313685    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adc39b46ea18d9526641cd25782340178787a8a4541835a41442e601a213d729\": container with ID starting with adc39b46ea18d9526641cd25782340178787a8a4541835a41442e601a213d729 not found: ID does not exist" containerID="adc39b46ea18d9526641cd25782340178787a8a4541835a41442e601a213d729"
Jul 22 02:56:10.313758 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.313730    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adc39b46ea18d9526641cd25782340178787a8a4541835a41442e601a213d729"} err="failed to get container status \"adc39b46ea18d9526641cd25782340178787a8a4541835a41442e601a213d729\": rpc error: code = NotFound desc = could not find container \"adc39b46ea18d9526641cd25782340178787a8a4541835a41442e601a213d729\": container with ID starting with adc39b46ea18d9526641cd25782340178787a8a4541835a41442e601a213d729 not found: ID does not exist"
Jul 22 02:56:10.313758 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.313747    2565 scope.go:117] "RemoveContainer" containerID="1c59f713f8ada509674737345b7894e72f73a2f2e54aa4f27cc42eed26e58db3"
Jul 22 02:56:10.313968 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:56:10.313951    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c59f713f8ada509674737345b7894e72f73a2f2e54aa4f27cc42eed26e58db3\": container with ID starting with 1c59f713f8ada509674737345b7894e72f73a2f2e54aa4f27cc42eed26e58db3 not found: ID does not exist" containerID="1c59f713f8ada509674737345b7894e72f73a2f2e54aa4f27cc42eed26e58db3"
Jul 22 02:56:10.314019 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.313974    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c59f713f8ada509674737345b7894e72f73a2f2e54aa4f27cc42eed26e58db3"} err="failed to get container status \"1c59f713f8ada509674737345b7894e72f73a2f2e54aa4f27cc42eed26e58db3\": rpc error: code = NotFound desc = could not find container \"1c59f713f8ada509674737345b7894e72f73a2f2e54aa4f27cc42eed26e58db3\": container with ID starting with 1c59f713f8ada509674737345b7894e72f73a2f2e54aa4f27cc42eed26e58db3 not found: ID does not exist"
Jul 22 02:56:10.323125 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.323097    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"]
Jul 22 02:56:10.327426 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:10.327406    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bd920c0c-openshift-default-76bbc947cf-sfmjx"]
Jul 22 02:56:11.726222 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:11.726190    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="895f9525-6d94-4801-9e0a-1897d0e38861" path="/var/lib/kubelet/pods/895f9525-6d94-4801-9e0a-1897d0e38861/volumes"
Jul 22 02:56:15.536327 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:15.536298    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:56:15.633638 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:15.633583    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:56:16.309133 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:16.309102    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:56:22.503535 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:22.503501    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:56:22.610832 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:22.610795    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:56:22.704300 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:22.704266    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:56:33.249183 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:33.249150    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:56:33.615543 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:33.615514    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:56:54.610499 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:56:54.610464    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:00.691406 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:00.691370    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:00.799305 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:00.799275    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:01.053878 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.053772    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:01.365601 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.365567    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"]
Jul 22 02:57:01.365991 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.365976    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f84e6d5-c430-4378-b86b-901c90231a84" containerName="main"
Jul 22 02:57:01.366042 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.365992    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f84e6d5-c430-4378-b86b-901c90231a84" containerName="main"
Jul 22 02:57:01.366042 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.366007    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="895f9525-6d94-4801-9e0a-1897d0e38861" containerName="istio-proxy"
Jul 22 02:57:01.366042 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.366012    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="895f9525-6d94-4801-9e0a-1897d0e38861" containerName="istio-proxy"
Jul 22 02:57:01.366042 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.366027    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64a6d693-99aa-4e57-9fbd-f8058283c758" containerName="maas-api"
Jul 22 02:57:01.366042 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.366032    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="64a6d693-99aa-4e57-9fbd-f8058283c758" containerName="maas-api"
Jul 22 02:57:01.366205 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.366084    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="895f9525-6d94-4801-9e0a-1897d0e38861" containerName="istio-proxy"
Jul 22 02:57:01.366205 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.366091    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f84e6d5-c430-4378-b86b-901c90231a84" containerName="main"
Jul 22 02:57:01.366205 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.366097    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="64a6d693-99aa-4e57-9fbd-f8058283c758" containerName="maas-api"
Jul 22 02:57:01.366205 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.366103    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="895f9525-6d94-4801-9e0a-1897d0e38861" containerName="istio-proxy"
Jul 22 02:57:01.369237 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.369218    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.371662 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:57:01.371623    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="6b6fb240-4df1-45d9-8b43-319639d4087f" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.372371 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:57:01.372344    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-inf-a-187dc9-openshift-default-dockercfg-gbdr9\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-187dc9-openshift-default-dockercfg-gbdr9\"" type="*v1.Secret"
Jul 22 02:57:01.396006 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.395978    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:01.397113 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.397090    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"]
Jul 22 02:57:01.417978 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.417955    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-credential-socket\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.418091 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.417987    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-token\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.418091 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.418007    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-workload-certs\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.418091 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.418033    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-workload-socket\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.418091 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.418055    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-data\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.418347 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.418112    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-envoy\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.418347 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.418142    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j5tw\" (UniqueName: \"kubernetes.io/projected/6b6fb240-4df1-45d9-8b43-319639d4087f-kube-api-access-5j5tw\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.418347 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.418165    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6fb240-4df1-45d9-8b43-319639d4087f-istiod-ca-cert\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.418347 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.418258    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-podinfo\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.519204 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.519170    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-podinfo\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.519409 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.519230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-credential-socket\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.519409 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.519261    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-token\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.519409 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.519287    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-workload-certs\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.519409 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.519320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-workload-socket\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.519409 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.519342    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-data\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.519409 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.519373    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-envoy\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.519409 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.519406    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j5tw\" (UniqueName: \"kubernetes.io/projected/6b6fb240-4df1-45d9-8b43-319639d4087f-kube-api-access-5j5tw\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.519776 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.519441    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6fb240-4df1-45d9-8b43-319639d4087f-istiod-ca-cert\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.519776 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.519739    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-credential-socket\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.519934 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.519894    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-workload-socket\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.520192 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.520165    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6fb240-4df1-45d9-8b43-319639d4087f-istiod-ca-cert\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.520331 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.520170    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-workload-certs\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.520398 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.520380    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-data\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.522150 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.522127    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-podinfo\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.522375 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.522358    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-envoy\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.533134 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.533112    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-token\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.546247 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.546214    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j5tw\" (UniqueName: \"kubernetes.io/projected/6b6fb240-4df1-45d9-8b43-319639d4087f-kube-api-access-5j5tw\") pod \"e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") " pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:01.620641 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:01.620573    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:02.198977 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:02.198944    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-187dc9-openshift-default-dockercfg-gbdr9\""
Jul 22 02:57:02.199955 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:02.199938    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:02.345565 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:02.345493    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"]
Jul 22 02:57:02.347801 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:57:02.347778    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6fb240_4df1_45d9_8b43_319639d4087f.slice/crio-4da46519d2d3eb50fb6fc551a9e8498867b833ce30fab8c293bd292484169b0c WatchSource:0}: Error finding container 4da46519d2d3eb50fb6fc551a9e8498867b833ce30fab8c293bd292484169b0c: Status 404 returned error can't find the container with id 4da46519d2d3eb50fb6fc551a9e8498867b833ce30fab8c293bd292484169b0c
Jul 22 02:57:02.349935 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:02.349902    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:57:02.350030 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:02.349968    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:57:02.350030 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:02.349996    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:57:02.531439 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:02.531359    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv" event={"ID":"6b6fb240-4df1-45d9-8b43-319639d4087f","Type":"ContainerStarted","Data":"db2bd733de7f1618419a3589c1769668042a0296d3c7900ff906362e06ba2953"}
Jul 22 02:57:02.531439 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:02.531432    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv" event={"ID":"6b6fb240-4df1-45d9-8b43-319639d4087f","Type":"ContainerStarted","Data":"4da46519d2d3eb50fb6fc551a9e8498867b833ce30fab8c293bd292484169b0c"}
Jul 22 02:57:02.557860 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:02.557792    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv" podStartSLOduration=1.5577759310000001 podStartE2EDuration="1.557775931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:57:02.55539108 +0000 UTC m=+2173.403727654" watchObservedRunningTime="2026-07-22 02:57:02.557775931 +0000 UTC m=+2173.406112505"
Jul 22 02:57:03.200281 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:03.200250    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:03.205081 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:03.205050    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:03.536143 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:03.536070    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:03.536926 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:03.536907    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:04.543598 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:04.543566    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:04.645242 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:04.645210    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:04.774463 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:04.774428    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"]
Jul 22 02:57:04.774826 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:04.774810    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="895f9525-6d94-4801-9e0a-1897d0e38861" containerName="istio-proxy"
Jul 22 02:57:04.774826 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:04.774823    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="895f9525-6d94-4801-9e0a-1897d0e38861" containerName="istio-proxy"
Jul 22 02:57:04.778107 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:04.778088    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"
Jul 22 02:57:04.780735 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:04.780712    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-187dc9-serving-cert\""
Jul 22 02:57:04.791830 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:04.791801    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"]
Jul 22 02:57:04.809064 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:04.809006    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:04.848085 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:04.848053    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdsc6\" (UniqueName: \"kubernetes.io/projected/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-kube-api-access-gdsc6\") pod \"maas-api-e2e-inf-a-187dc9-99b567b69-799bn\" (UID: \"9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"
Jul 22 02:57:04.848215 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:04.848171    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-187dc9-99b567b69-799bn\" (UID: \"9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"
Jul 22 02:57:04.848280 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:04.848234    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-187dc9-99b567b69-799bn\" (UID: \"9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"
Jul 22 02:57:04.949073 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:04.949033    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-187dc9-99b567b69-799bn\" (UID: \"9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"
Jul 22 02:57:04.949223 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:04.949088    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-187dc9-99b567b69-799bn\" (UID: \"9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"
Jul 22 02:57:04.949223 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:04.949134    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdsc6\" (UniqueName: \"kubernetes.io/projected/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-kube-api-access-gdsc6\") pod \"maas-api-e2e-inf-a-187dc9-99b567b69-799bn\" (UID: \"9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"
Jul 22 02:57:04.949298 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:57:04.949239    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-serving-cert: secret "maas-api-e2e-inf-a-187dc9-serving-cert" not found
Jul 22 02:57:04.949337 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:57:04.949303    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-maas-api-tls podName:9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa nodeName:}" failed. No retries permitted until 2026-07-22 02:57:05.449287189 +0000 UTC m=+2176.297623741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-maas-api-tls") pod "maas-api-e2e-inf-a-187dc9-99b567b69-799bn" (UID: "9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa") : secret "maas-api-e2e-inf-a-187dc9-serving-cert" not found
Jul 22 02:57:04.949667 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:04.949648    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-187dc9-99b567b69-799bn\" (UID: \"9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"
Jul 22 02:57:04.968658 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:04.968633    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdsc6\" (UniqueName: \"kubernetes.io/projected/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-kube-api-access-gdsc6\") pod \"maas-api-e2e-inf-a-187dc9-99b567b69-799bn\" (UID: \"9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"
Jul 22 02:57:05.127878 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:05.127822    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:05.309822 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:05.309794    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:05.416961 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:05.416884    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:05.454390 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:05.454354    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-187dc9-99b567b69-799bn\" (UID: \"9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"
Jul 22 02:57:05.457078 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:05.457052    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-187dc9-99b567b69-799bn\" (UID: \"9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"
Jul 22 02:57:05.689144 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:05.689052    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"
Jul 22 02:57:05.836037 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:05.835869    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"]
Jul 22 02:57:05.838327 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:57:05.838301    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c97d41e_e2a8_4f9c_b03f_96e5fc70d9aa.slice/crio-7a3577dadee086af001487f8e88d276513aeb55005c03b25aa8069f4923a8b92 WatchSource:0}: Error finding container 7a3577dadee086af001487f8e88d276513aeb55005c03b25aa8069f4923a8b92: Status 404 returned error can't find the container with id 7a3577dadee086af001487f8e88d276513aeb55005c03b25aa8069f4923a8b92
Jul 22 02:57:06.551893 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:06.551838    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn" event={"ID":"9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa","Type":"ContainerStarted","Data":"7a3577dadee086af001487f8e88d276513aeb55005c03b25aa8069f4923a8b92"}
Jul 22 02:57:07.558216 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:07.558182    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn" event={"ID":"9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa","Type":"ContainerStarted","Data":"13a6c5b9167cea8780dc911e5c96623ea3f47ac452632476fee0faad14e02467"}
Jul 22 02:57:07.558690 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:07.558307    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"
Jul 22 02:57:07.576469 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:07.576426    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn" podStartSLOduration=2.858729264 podStartE2EDuration="3.576412819s" totalImagesPullingTime="717.683555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:57:07.575126728 +0000 UTC m=+2178.423463303" watchObservedRunningTime="2026-07-22 02:57:07.576412819 +0000 UTC m=+2178.424749392"
Jul 22 02:57:09.021931 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:09.021895    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:09.382020 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:09.381982    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:09.715185 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:09.715093    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:10.091171 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:10.091079    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:13.174043 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.173989    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:13.250403 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.250371    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"]
Jul 22 02:57:13.254014 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.253993    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"
Jul 22 02:57:13.256957 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.256935    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6ecc40-serving-cert\""
Jul 22 02:57:13.267034 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.265973    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"]
Jul 22 02:57:13.272260 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.272239    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:13.331868 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.331440    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e31d7313-7a93-425d-b295-b2d385a411c9-maas-api-tls\") pod \"maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5\" (UID: \"e31d7313-7a93-425d-b295-b2d385a411c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"
Jul 22 02:57:13.331868 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.331537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w67m7\" (UniqueName: \"kubernetes.io/projected/e31d7313-7a93-425d-b295-b2d385a411c9-kube-api-access-w67m7\") pod \"maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5\" (UID: \"e31d7313-7a93-425d-b295-b2d385a411c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"
Jul 22 02:57:13.331868 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.331621    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31d7313-7a93-425d-b295-b2d385a411c9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5\" (UID: \"e31d7313-7a93-425d-b295-b2d385a411c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"
Jul 22 02:57:13.345493 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.345461    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:13.432751 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.432643    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e31d7313-7a93-425d-b295-b2d385a411c9-maas-api-tls\") pod \"maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5\" (UID: \"e31d7313-7a93-425d-b295-b2d385a411c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"
Jul 22 02:57:13.432751 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.432734    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w67m7\" (UniqueName: \"kubernetes.io/projected/e31d7313-7a93-425d-b295-b2d385a411c9-kube-api-access-w67m7\") pod \"maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5\" (UID: \"e31d7313-7a93-425d-b295-b2d385a411c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"
Jul 22 02:57:13.433003 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.432795    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31d7313-7a93-425d-b295-b2d385a411c9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5\" (UID: \"e31d7313-7a93-425d-b295-b2d385a411c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"
Jul 22 02:57:13.433638 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.433606    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31d7313-7a93-425d-b295-b2d385a411c9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5\" (UID: \"e31d7313-7a93-425d-b295-b2d385a411c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"
Jul 22 02:57:13.433766 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:57:13.433735    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-serving-cert: secret "maas-api-e2e-inf-b-6ecc40-serving-cert" not found
Jul 22 02:57:13.433830 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:57:13.433796    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e31d7313-7a93-425d-b295-b2d385a411c9-maas-api-tls podName:e31d7313-7a93-425d-b295-b2d385a411c9 nodeName:}" failed. No retries permitted until 2026-07-22 02:57:13.933776192 +0000 UTC m=+2184.782112744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e31d7313-7a93-425d-b295-b2d385a411c9-maas-api-tls") pod "maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5" (UID: "e31d7313-7a93-425d-b295-b2d385a411c9") : secret "maas-api-e2e-inf-b-6ecc40-serving-cert" not found
Jul 22 02:57:13.444928 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.444902    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w67m7\" (UniqueName: \"kubernetes.io/projected/e31d7313-7a93-425d-b295-b2d385a411c9-kube-api-access-w67m7\") pod \"maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5\" (UID: \"e31d7313-7a93-425d-b295-b2d385a411c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"
Jul 22 02:57:13.568950 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.568922    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"
Jul 22 02:57:13.662071 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.662037    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:13.859190 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.859158    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:13.940959 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.940923    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e31d7313-7a93-425d-b295-b2d385a411c9-maas-api-tls\") pod \"maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5\" (UID: \"e31d7313-7a93-425d-b295-b2d385a411c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"
Jul 22 02:57:13.943467 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.943432    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e31d7313-7a93-425d-b295-b2d385a411c9-maas-api-tls\") pod \"maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5\" (UID: \"e31d7313-7a93-425d-b295-b2d385a411c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"
Jul 22 02:57:13.959031 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:13.959008    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:14.168377 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:14.168287    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"
Jul 22 02:57:14.333693 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:14.328120    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"]
Jul 22 02:57:14.340445 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:57:14.340418    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode31d7313_7a93_425d_b295_b2d385a411c9.slice/crio-d173a9dc3a6d20049a9f1db46590d36360133021a76e3a411a843746a9018244 WatchSource:0}: Error finding container d173a9dc3a6d20049a9f1db46590d36360133021a76e3a411a843746a9018244: Status 404 returned error can't find the container with id d173a9dc3a6d20049a9f1db46590d36360133021a76e3a411a843746a9018244
Jul 22 02:57:14.592744 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:14.592710    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5" event={"ID":"e31d7313-7a93-425d-b295-b2d385a411c9","Type":"ContainerStarted","Data":"d173a9dc3a6d20049a9f1db46590d36360133021a76e3a411a843746a9018244"}
Jul 22 02:57:15.599262 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:15.599228    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5" event={"ID":"e31d7313-7a93-425d-b295-b2d385a411c9","Type":"ContainerStarted","Data":"8cea0d43f53f283e3ccc0ceed65f45753b101e6a3b0e0479ec28ee45eab9a7a8"}
Jul 22 02:57:15.599639 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:15.599367    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"
Jul 22 02:57:15.619575 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:15.619517    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5" podStartSLOduration=2.2692035280000002 podStartE2EDuration="2.619501415s" totalImagesPullingTime="350.297887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:57:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:57:15.618199357 +0000 UTC m=+2186.466535931" watchObservedRunningTime="2026-07-22 02:57:15.619501415 +0000 UTC m=+2186.467837988"
Jul 22 02:57:17.610748 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:17.610715    2565 generic.go:356] "Generic (PLEG): container finished" podID="6b6fb240-4df1-45d9-8b43-319639d4087f" containerID="db2bd733de7f1618419a3589c1769668042a0296d3c7900ff906362e06ba2953" exitCode=0
Jul 22 02:57:17.611158 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:17.610791    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv" event={"ID":"6b6fb240-4df1-45d9-8b43-319639d4087f","Type":"ContainerDied","Data":"db2bd733de7f1618419a3589c1769668042a0296d3c7900ff906362e06ba2953"}
Jul 22 02:57:17.611214 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:17.611201    2565 scope.go:117] "RemoveContainer" containerID="db2bd733de7f1618419a3589c1769668042a0296d3c7900ff906362e06ba2953"
Jul 22 02:57:17.612269 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:17.612243    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:57:17.612342 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:17.612311    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:57:17.612381 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:17.612343    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:57:17.695000 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:17.694969    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:18.536780 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:18.536745    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:18.621875 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:18.620978    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv" event={"ID":"6b6fb240-4df1-45d9-8b43-319639d4087f","Type":"ContainerStarted","Data":"01b1933230b2ad9af453022cc64b6b643c9793b8a3f5c8bfc5d8b88cb88318dc"}
Jul 22 02:57:19.200800 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:19.200763    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:19.201789 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:19.201764    2565 patch_prober.go:28] interesting pod/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 22 02:57:19.201924 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:19.201811    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv" podUID="6b6fb240-4df1-45d9-8b43-319639d4087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 22 02:57:19.520070 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:19.519943    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:20.200905 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:20.200839    2565 patch_prober.go:28] interesting pod/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 22 02:57:20.201356 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:20.200926    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv" podUID="6b6fb240-4df1-45d9-8b43-319639d4087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 22 02:57:21.200866 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:21.200824    2565 patch_prober.go:28] interesting pod/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 22 02:57:21.201113 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:21.200905    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv" podUID="6b6fb240-4df1-45d9-8b43-319639d4087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 22 02:57:21.609994 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:21.609966    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"
Jul 22 02:57:22.200906 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:22.200872    2565 patch_prober.go:28] interesting pod/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 22 02:57:22.201093 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:22.200927    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv" podUID="6b6fb240-4df1-45d9-8b43-319639d4087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 22 02:57:23.204367 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:23.204336    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:23.204741 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:23.204574    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:23.205221 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:23.205204    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:57:26.967881 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:26.967782    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:27.074033 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:27.074004    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:31.112838 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:31.112801    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:34.646776 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:34.646737    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:34.747089 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:34.747053    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:47.375719 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.375676    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"]
Jul 22 02:57:47.378597 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.378575    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:47.382251 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.382222    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6ecc40\"/\"openshift-service-ca.crt\""
Jul 22 02:57:47.382438 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.382383    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6ecc40\"/\"test-model-6ecc40-kserve-self-signed-certs\""
Jul 22 02:57:47.382523 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.382352    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6ecc40\"/\"kube-root-ca.crt\""
Jul 22 02:57:47.382620 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.382593    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6ecc40\"/\"default-dockercfg-29qkc\""
Jul 22 02:57:47.388332 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.388306    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"]
Jul 22 02:57:47.416237 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.416216    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:47.419403 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.419372    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f0c78a-3af2-4071-99bc-c908260df53e-tls-certs\") pod \"test-model-6ecc40-kserve-54b5c76587-54l69\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") " pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:47.419509 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.419446    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5w2z\" (UniqueName: \"kubernetes.io/projected/05f0c78a-3af2-4071-99bc-c908260df53e-kube-api-access-b5w2z\") pod \"test-model-6ecc40-kserve-54b5c76587-54l69\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") " pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:47.419509 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.419481    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-tmp-dir\") pod \"test-model-6ecc40-kserve-54b5c76587-54l69\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") " pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:47.419623 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.419510    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-home\") pod \"test-model-6ecc40-kserve-54b5c76587-54l69\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") " pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:47.419623 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.419566    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-dshm\") pod \"test-model-6ecc40-kserve-54b5c76587-54l69\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") " pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:47.419623 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.419593    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-model-cache\") pod \"test-model-6ecc40-kserve-54b5c76587-54l69\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") " pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:47.520978 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.520948    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5w2z\" (UniqueName: \"kubernetes.io/projected/05f0c78a-3af2-4071-99bc-c908260df53e-kube-api-access-b5w2z\") pod \"test-model-6ecc40-kserve-54b5c76587-54l69\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") " pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:47.521136 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.520989    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-tmp-dir\") pod \"test-model-6ecc40-kserve-54b5c76587-54l69\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") " pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:47.521136 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.521013    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-home\") pod \"test-model-6ecc40-kserve-54b5c76587-54l69\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") " pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:47.521257 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.521127    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-dshm\") pod \"test-model-6ecc40-kserve-54b5c76587-54l69\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") " pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:47.521257 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.521159    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-model-cache\") pod \"test-model-6ecc40-kserve-54b5c76587-54l69\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") " pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:47.521353 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.521299    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f0c78a-3af2-4071-99bc-c908260df53e-tls-certs\") pod \"test-model-6ecc40-kserve-54b5c76587-54l69\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") " pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:47.521412 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.521386    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-home\") pod \"test-model-6ecc40-kserve-54b5c76587-54l69\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") " pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:47.521459 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.521417    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-tmp-dir\") pod \"test-model-6ecc40-kserve-54b5c76587-54l69\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") " pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:47.521502 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.521460    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-model-cache\") pod \"test-model-6ecc40-kserve-54b5c76587-54l69\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") " pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:47.523250 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.523230    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-dshm\") pod \"test-model-6ecc40-kserve-54b5c76587-54l69\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") " pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:47.523548 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.523530    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f0c78a-3af2-4071-99bc-c908260df53e-tls-certs\") pod \"test-model-6ecc40-kserve-54b5c76587-54l69\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") " pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:47.531911 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.531889    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5w2z\" (UniqueName: \"kubernetes.io/projected/05f0c78a-3af2-4071-99bc-c908260df53e-kube-api-access-b5w2z\") pod \"test-model-6ecc40-kserve-54b5c76587-54l69\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") " pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:47.689388 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.689306    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:47.823016 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:47.822985    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"]
Jul 22 02:57:47.824940 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:57:47.824911    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05f0c78a_3af2_4071_99bc_c908260df53e.slice/crio-a030af6791e4c38108d08bc12855b702df7bf4ac7bef5f78be7d3a6dc50cf2be WatchSource:0}: Error finding container a030af6791e4c38108d08bc12855b702df7bf4ac7bef5f78be7d3a6dc50cf2be: Status 404 returned error can't find the container with id a030af6791e4c38108d08bc12855b702df7bf4ac7bef5f78be7d3a6dc50cf2be
Jul 22 02:57:48.766811 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:48.766771    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69" event={"ID":"05f0c78a-3af2-4071-99bc-c908260df53e","Type":"ContainerStarted","Data":"c7c2d30d2086f4bdcb42a700f516efc4356af1efa32b24a966d54ca3068e6a68"}
Jul 22 02:57:48.766811 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:48.766813    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69" event={"ID":"05f0c78a-3af2-4071-99bc-c908260df53e","Type":"ContainerStarted","Data":"a030af6791e4c38108d08bc12855b702df7bf4ac7bef5f78be7d3a6dc50cf2be"}
Jul 22 02:57:48.786357 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:48.786307    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69" podStartSLOduration=1.7862910090000002 podStartE2EDuration="1.786291009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:57:48.784463438 +0000 UTC m=+2219.632800011" watchObservedRunningTime="2026-07-22 02:57:48.786291009 +0000 UTC m=+2219.634627584"
Jul 22 02:57:50.077702 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:50.077666    2565 scope.go:117] "RemoveContainer" containerID="28acbcfb16816b483d14a66222f5a659dac4c91447986641a2e4b8a7b5b57832"
Jul 22 02:57:57.690486 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:57.690453    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:57.691094 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:57.690508    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:57.702546 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:57.702524    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:57.823338 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:57.823310    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:57:58.512387 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:58.512346    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:57:59.843920 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:57:59.843886    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:58:26.880770 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:26.880740    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:58:27.765833 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:27.765803    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"]
Jul 22 02:58:27.787223 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:27.787197    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:58:28.524088 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:28.524051    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:58:28.630614 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:28.630583    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:58:38.379781 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:38.379748    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"]
Jul 22 02:58:38.380261 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:38.380012    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn" podUID="9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa" containerName="maas-api" containerID="cri-o://13a6c5b9167cea8780dc911e5c96623ea3f47ac452632476fee0faad14e02467" gracePeriod=30
Jul 22 02:58:38.412256 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:38.412229    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:58:38.564070 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:38.564030    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn" podUID="9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.57:8443/health\": dial tcp 10.133.0.57:8443: connect: connection refused"
Jul 22 02:58:38.830692 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:38.830664    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"
Jul 22 02:58:38.880140 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:38.880108    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdsc6\" (UniqueName: \"kubernetes.io/projected/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-kube-api-access-gdsc6\") pod \"9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa\" (UID: \"9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa\") "
Jul 22 02:58:38.880321 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:38.880167    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-maas-api-tls\") pod \"9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa\" (UID: \"9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa\") "
Jul 22 02:58:38.880321 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:38.880193    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-openshift-service-ca\") pod \"9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa\" (UID: \"9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa\") "
Jul 22 02:58:38.880624 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:38.880590    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa" (UID: "9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:58:38.882238 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:38.882208    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-kube-api-access-gdsc6" (OuterVolumeSpecName: "kube-api-access-gdsc6") pod "9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa" (UID: "9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa"). InnerVolumeSpecName "kube-api-access-gdsc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:58:38.882313 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:38.882270    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa" (UID: "9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:58:38.981293 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:38.981221    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdsc6\" (UniqueName: \"kubernetes.io/projected/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-kube-api-access-gdsc6\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:58:38.981293 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:38.981248    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:58:38.981293 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:38.981257    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:58:39.000979 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:39.000952    2565 generic.go:356] "Generic (PLEG): container finished" podID="9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa" containerID="13a6c5b9167cea8780dc911e5c96623ea3f47ac452632476fee0faad14e02467" exitCode=0
Jul 22 02:58:39.001100 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:39.001001    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"
Jul 22 02:58:39.001100 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:39.001044    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn" event={"ID":"9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa","Type":"ContainerDied","Data":"13a6c5b9167cea8780dc911e5c96623ea3f47ac452632476fee0faad14e02467"}
Jul 22 02:58:39.001100 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:39.001091    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn" event={"ID":"9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa","Type":"ContainerDied","Data":"7a3577dadee086af001487f8e88d276513aeb55005c03b25aa8069f4923a8b92"}
Jul 22 02:58:39.001212 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:39.001117    2565 scope.go:117] "RemoveContainer" containerID="13a6c5b9167cea8780dc911e5c96623ea3f47ac452632476fee0faad14e02467"
Jul 22 02:58:39.012439 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:39.012417    2565 scope.go:117] "RemoveContainer" containerID="13a6c5b9167cea8780dc911e5c96623ea3f47ac452632476fee0faad14e02467"
Jul 22 02:58:39.012703 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:58:39.012684    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a6c5b9167cea8780dc911e5c96623ea3f47ac452632476fee0faad14e02467\": container with ID starting with 13a6c5b9167cea8780dc911e5c96623ea3f47ac452632476fee0faad14e02467 not found: ID does not exist" containerID="13a6c5b9167cea8780dc911e5c96623ea3f47ac452632476fee0faad14e02467"
Jul 22 02:58:39.012772 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:39.012718    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a6c5b9167cea8780dc911e5c96623ea3f47ac452632476fee0faad14e02467"} err="failed to get container status \"13a6c5b9167cea8780dc911e5c96623ea3f47ac452632476fee0faad14e02467\": rpc error: code = NotFound desc = could not find container \"13a6c5b9167cea8780dc911e5c96623ea3f47ac452632476fee0faad14e02467\": container with ID starting with 13a6c5b9167cea8780dc911e5c96623ea3f47ac452632476fee0faad14e02467 not found: ID does not exist"
Jul 22 02:58:39.025074 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:39.025046    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"]
Jul 22 02:58:39.029803 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:39.029784    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-187dc9-99b567b69-799bn"]
Jul 22 02:58:39.725502 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:39.725468    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa" path="/var/lib/kubelet/pods/9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa/volumes"
Jul 22 02:58:42.786372 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:42.786335    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69" podUID="05f0c78a-3af2-4071-99bc-c908260df53e" containerName="main" containerID="cri-o://c7c2d30d2086f4bdcb42a700f516efc4356af1efa32b24a966d54ca3068e6a68" gracePeriod=45
Jul 22 02:58:43.008613 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.008581    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:58:43.021603 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.021576    2565 generic.go:356] "Generic (PLEG): container finished" podID="05f0c78a-3af2-4071-99bc-c908260df53e" containerID="c7c2d30d2086f4bdcb42a700f516efc4356af1efa32b24a966d54ca3068e6a68" exitCode=0
Jul 22 02:58:43.021731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.021631    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69" event={"ID":"05f0c78a-3af2-4071-99bc-c908260df53e","Type":"ContainerDied","Data":"c7c2d30d2086f4bdcb42a700f516efc4356af1efa32b24a966d54ca3068e6a68"}
Jul 22 02:58:43.021731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.021659    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69" event={"ID":"05f0c78a-3af2-4071-99bc-c908260df53e","Type":"ContainerDied","Data":"a030af6791e4c38108d08bc12855b702df7bf4ac7bef5f78be7d3a6dc50cf2be"}
Jul 22 02:58:43.021731 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.021671    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a030af6791e4c38108d08bc12855b702df7bf4ac7bef5f78be7d3a6dc50cf2be"
Jul 22 02:58:43.036156 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.036135    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:58:43.116108 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.116085    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-dshm\") pod \"05f0c78a-3af2-4071-99bc-c908260df53e\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") "
Jul 22 02:58:43.116249 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.116148    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f0c78a-3af2-4071-99bc-c908260df53e-tls-certs\") pod \"05f0c78a-3af2-4071-99bc-c908260df53e\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") "
Jul 22 02:58:43.116249 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.116167    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-model-cache\") pod \"05f0c78a-3af2-4071-99bc-c908260df53e\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") "
Jul 22 02:58:43.116249 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.116184    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5w2z\" (UniqueName: \"kubernetes.io/projected/05f0c78a-3af2-4071-99bc-c908260df53e-kube-api-access-b5w2z\") pod \"05f0c78a-3af2-4071-99bc-c908260df53e\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") "
Jul 22 02:58:43.116249 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.116199    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-home\") pod \"05f0c78a-3af2-4071-99bc-c908260df53e\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") "
Jul 22 02:58:43.116249 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.116233    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-tmp-dir\") pod \"05f0c78a-3af2-4071-99bc-c908260df53e\" (UID: \"05f0c78a-3af2-4071-99bc-c908260df53e\") "
Jul 22 02:58:43.116517 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.116419    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-model-cache" (OuterVolumeSpecName: "model-cache") pod "05f0c78a-3af2-4071-99bc-c908260df53e" (UID: "05f0c78a-3af2-4071-99bc-c908260df53e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:58:43.116584 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.116513    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-home" (OuterVolumeSpecName: "home") pod "05f0c78a-3af2-4071-99bc-c908260df53e" (UID: "05f0c78a-3af2-4071-99bc-c908260df53e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:58:43.116584 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.116525    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-model-cache\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:58:43.116584 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.116549    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "05f0c78a-3af2-4071-99bc-c908260df53e" (UID: "05f0c78a-3af2-4071-99bc-c908260df53e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:58:43.118407 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.118380    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f0c78a-3af2-4071-99bc-c908260df53e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05f0c78a-3af2-4071-99bc-c908260df53e" (UID: "05f0c78a-3af2-4071-99bc-c908260df53e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:58:43.118513 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.118437    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-dshm" (OuterVolumeSpecName: "dshm") pod "05f0c78a-3af2-4071-99bc-c908260df53e" (UID: "05f0c78a-3af2-4071-99bc-c908260df53e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:58:43.118513 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.118465    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f0c78a-3af2-4071-99bc-c908260df53e-kube-api-access-b5w2z" (OuterVolumeSpecName: "kube-api-access-b5w2z") pod "05f0c78a-3af2-4071-99bc-c908260df53e" (UID: "05f0c78a-3af2-4071-99bc-c908260df53e"). InnerVolumeSpecName "kube-api-access-b5w2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:58:43.217226 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.217187    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f0c78a-3af2-4071-99bc-c908260df53e-tls-certs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:58:43.217226 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.217225    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5w2z\" (UniqueName: \"kubernetes.io/projected/05f0c78a-3af2-4071-99bc-c908260df53e-kube-api-access-b5w2z\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:58:43.217439 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.217240    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-home\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:58:43.217439 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.217252    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-tmp-dir\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:58:43.217439 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.217265    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05f0c78a-3af2-4071-99bc-c908260df53e-dshm\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:58:43.449738 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:43.449660    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:58:44.026165 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:44.026087    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"
Jul 22 02:58:44.041897 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:44.041870    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"]
Jul 22 02:58:44.045832 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:44.045797    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6ecc40/test-model-6ecc40-kserve-54b5c76587-54l69"]
Jul 22 02:58:45.725396 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:45.725365    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05f0c78a-3af2-4071-99bc-c908260df53e" path="/var/lib/kubelet/pods/05f0c78a-3af2-4071-99bc-c908260df53e/volumes"
Jul 22 02:58:49.555491 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:49.555456    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:58:49.675177 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:49.675135    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:58:49.869838 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:49.869807    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"]
Jul 22 02:58:49.870063 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:49.870042    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv" podUID="6b6fb240-4df1-45d9-8b43-319639d4087f" containerName="istio-proxy" containerID="cri-o://01b1933230b2ad9af453022cc64b6b643c9793b8a3f5c8bfc5d8b88cb88318dc" gracePeriod=30
Jul 22 02:58:49.887663 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:49.887634    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:58:50.097993 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:50.097954    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:58:50.224908 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:50.224822    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ecc40-rknc8"]
Jul 22 02:58:50.225207 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:50.225193    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05f0c78a-3af2-4071-99bc-c908260df53e" containerName="main"
Jul 22 02:58:50.225207 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:50.225207    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="05f0c78a-3af2-4071-99bc-c908260df53e" containerName="main"
Jul 22 02:58:50.225330 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:50.225222    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa" containerName="maas-api"
Jul 22 02:58:50.225330 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:50.225227    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa" containerName="maas-api"
Jul 22 02:58:50.225330 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:50.225281    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c97d41e-e2a8-4f9c-b03f-96e5fc70d9aa" containerName="maas-api"
Jul 22 02:58:50.225330 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:50.225292    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="05f0c78a-3af2-4071-99bc-c908260df53e" containerName="main"
Jul 22 02:58:50.228049 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:50.228028    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ecc40-rknc8"
Jul 22 02:58:50.230677 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:50.230654    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vtlqd\""
Jul 22 02:58:50.235981 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:50.235956    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ecc40-rknc8"]
Jul 22 02:58:50.279067 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:50.279035    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219c8f78-be9f-4086-baeb-976a1d7c3c64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6ecc40-rknc8\" (UID: \"219c8f78-be9f-4086-baeb-976a1d7c3c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ecc40-rknc8"
Jul 22 02:58:50.379817 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:50.379784    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219c8f78-be9f-4086-baeb-976a1d7c3c64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6ecc40-rknc8\" (UID: \"219c8f78-be9f-4086-baeb-976a1d7c3c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ecc40-rknc8"
Jul 22 02:58:50.380405 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:50.380386    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219c8f78-be9f-4086-baeb-976a1d7c3c64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6ecc40-rknc8\" (UID: \"219c8f78-be9f-4086-baeb-976a1d7c3c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ecc40-rknc8"
Jul 22 02:58:50.539983 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:50.539910    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ecc40-rknc8"
Jul 22 02:58:50.668722 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:50.668697    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ecc40-rknc8"]
Jul 22 02:58:50.670860 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:58:50.670819    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod219c8f78_be9f_4086_baeb_976a1d7c3c64.slice/crio-5415aa728240ff759a0ddfc0601fc1197a022aab684d7c0af30a893352228d18 WatchSource:0}: Error finding container 5415aa728240ff759a0ddfc0601fc1197a022aab684d7c0af30a893352228d18: Status 404 returned error can't find the container with id 5415aa728240ff759a0ddfc0601fc1197a022aab684d7c0af30a893352228d18
Jul 22 02:58:50.881861 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:50.881820    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:58:50.982446 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:50.982421    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:58:51.060372 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:51.060326    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ecc40-rknc8" event={"ID":"219c8f78-be9f-4086-baeb-976a1d7c3c64","Type":"ContainerStarted","Data":"ce58f459e4283576b323a1078998226aa38160545c1eafe1735cadd5c0b98324"}
Jul 22 02:58:51.060553 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:51.060383    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ecc40-rknc8" event={"ID":"219c8f78-be9f-4086-baeb-976a1d7c3c64","Type":"ContainerStarted","Data":"5415aa728240ff759a0ddfc0601fc1197a022aab684d7c0af30a893352228d18"}
Jul 22 02:58:51.076888 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:51.076806    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ecc40-rknc8" podStartSLOduration=1.076791567 podStartE2EDuration="1.076791567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:58:51.076092147 +0000 UTC m=+2281.924428732" watchObservedRunningTime="2026-07-22 02:58:51.076791567 +0000 UTC m=+2281.925128140"
Jul 22 02:58:52.067005 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:52.066966    2565 generic.go:356] "Generic (PLEG): container finished" podID="219c8f78-be9f-4086-baeb-976a1d7c3c64" containerID="ce58f459e4283576b323a1078998226aa38160545c1eafe1735cadd5c0b98324" exitCode=0
Jul 22 02:58:52.067390 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:52.067062    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ecc40-rknc8" event={"ID":"219c8f78-be9f-4086-baeb-976a1d7c3c64","Type":"ContainerDied","Data":"ce58f459e4283576b323a1078998226aa38160545c1eafe1735cadd5c0b98324"}
Jul 22 02:58:53.193244 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:53.193219    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ecc40-rknc8"
Jul 22 02:58:53.206652 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:53.206618    2565 patch_prober.go:28] interesting pod/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 22 02:58:53.206751 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:53.206664    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv" podUID="6b6fb240-4df1-45d9-8b43-319639d4087f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 22 02:58:53.307967 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:53.307925    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219c8f78-be9f-4086-baeb-976a1d7c3c64-maas-api-service-ca\") pod \"219c8f78-be9f-4086-baeb-976a1d7c3c64\" (UID: \"219c8f78-be9f-4086-baeb-976a1d7c3c64\") "
Jul 22 02:58:53.308263 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:53.308241    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/219c8f78-be9f-4086-baeb-976a1d7c3c64-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "219c8f78-be9f-4086-baeb-976a1d7c3c64" (UID: "219c8f78-be9f-4086-baeb-976a1d7c3c64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:58:53.409122 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:53.409084    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219c8f78-be9f-4086-baeb-976a1d7c3c64-maas-api-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:58:54.076442 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:54.076361    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ecc40-rknc8" event={"ID":"219c8f78-be9f-4086-baeb-976a1d7c3c64","Type":"ContainerDied","Data":"5415aa728240ff759a0ddfc0601fc1197a022aab684d7c0af30a893352228d18"}
Jul 22 02:58:54.076442 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:54.076411    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5415aa728240ff759a0ddfc0601fc1197a022aab684d7c0af30a893352228d18"
Jul 22 02:58:54.076442 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:54.076377    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ecc40-rknc8"
Jul 22 02:58:55.083684 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.083654    2565 generic.go:356] "Generic (PLEG): container finished" podID="6b6fb240-4df1-45d9-8b43-319639d4087f" containerID="01b1933230b2ad9af453022cc64b6b643c9793b8a3f5c8bfc5d8b88cb88318dc" exitCode=0
Jul 22 02:58:55.084093 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.083726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv" event={"ID":"6b6fb240-4df1-45d9-8b43-319639d4087f","Type":"ContainerDied","Data":"01b1933230b2ad9af453022cc64b6b643c9793b8a3f5c8bfc5d8b88cb88318dc"}
Jul 22 02:58:55.084093 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.083789    2565 scope.go:117] "RemoveContainer" containerID="db2bd733de7f1618419a3589c1769668042a0296d3c7900ff906362e06ba2953"
Jul 22 02:58:55.140799 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.140750    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:58:55.225675 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.225644    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-workload-socket\") pod \"6b6fb240-4df1-45d9-8b43-319639d4087f\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") "
Jul 22 02:58:55.225675 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.225672    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-envoy\") pod \"6b6fb240-4df1-45d9-8b43-319639d4087f\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") "
Jul 22 02:58:55.225909 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.225699    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-token\") pod \"6b6fb240-4df1-45d9-8b43-319639d4087f\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") "
Jul 22 02:58:55.225909 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.225716    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-data\") pod \"6b6fb240-4df1-45d9-8b43-319639d4087f\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") "
Jul 22 02:58:55.225909 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.225746    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-workload-certs\") pod \"6b6fb240-4df1-45d9-8b43-319639d4087f\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") "
Jul 22 02:58:55.225909 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.225785    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-podinfo\") pod \"6b6fb240-4df1-45d9-8b43-319639d4087f\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") "
Jul 22 02:58:55.225909 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.225815    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5j5tw\" (UniqueName: \"kubernetes.io/projected/6b6fb240-4df1-45d9-8b43-319639d4087f-kube-api-access-5j5tw\") pod \"6b6fb240-4df1-45d9-8b43-319639d4087f\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") "
Jul 22 02:58:55.225909 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.225864    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-credential-socket\") pod \"6b6fb240-4df1-45d9-8b43-319639d4087f\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") "
Jul 22 02:58:55.226274 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.225937    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6fb240-4df1-45d9-8b43-319639d4087f-istiod-ca-cert\") pod \"6b6fb240-4df1-45d9-8b43-319639d4087f\" (UID: \"6b6fb240-4df1-45d9-8b43-319639d4087f\") "
Jul 22 02:58:55.226274 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.225959    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6b6fb240-4df1-45d9-8b43-319639d4087f" (UID: "6b6fb240-4df1-45d9-8b43-319639d4087f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:58:55.226274 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.226228    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-workload-socket\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:58:55.226454 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.226430    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6b6fb240-4df1-45d9-8b43-319639d4087f" (UID: "6b6fb240-4df1-45d9-8b43-319639d4087f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:58:55.226558 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.226538    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6b6fb240-4df1-45d9-8b43-319639d4087f" (UID: "6b6fb240-4df1-45d9-8b43-319639d4087f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:58:55.226697 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.226672    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6fb240-4df1-45d9-8b43-319639d4087f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6b6fb240-4df1-45d9-8b43-319639d4087f" (UID: "6b6fb240-4df1-45d9-8b43-319639d4087f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:58:55.227165 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.227129    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-data" (OuterVolumeSpecName: "istio-data") pod "6b6fb240-4df1-45d9-8b43-319639d4087f" (UID: "6b6fb240-4df1-45d9-8b43-319639d4087f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:58:55.228161 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.228127    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6b6fb240-4df1-45d9-8b43-319639d4087f" (UID: "6b6fb240-4df1-45d9-8b43-319639d4087f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 22 02:58:55.228249 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.228182    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-token" (OuterVolumeSpecName: "istio-token") pod "6b6fb240-4df1-45d9-8b43-319639d4087f" (UID: "6b6fb240-4df1-45d9-8b43-319639d4087f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:58:55.228520 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.228499    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6b6fb240-4df1-45d9-8b43-319639d4087f" (UID: "6b6fb240-4df1-45d9-8b43-319639d4087f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:58:55.228602 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.228572    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6fb240-4df1-45d9-8b43-319639d4087f-kube-api-access-5j5tw" (OuterVolumeSpecName: "kube-api-access-5j5tw") pod "6b6fb240-4df1-45d9-8b43-319639d4087f" (UID: "6b6fb240-4df1-45d9-8b43-319639d4087f"). InnerVolumeSpecName "kube-api-access-5j5tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:58:55.326772 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.326747    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-envoy\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:58:55.326772 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.326768    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-token\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:58:55.326772 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.326777    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-data\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:58:55.327046 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.326786    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-workload-certs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:58:55.327046 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.326794    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6fb240-4df1-45d9-8b43-319639d4087f-istio-podinfo\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:58:55.327046 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.326802    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j5tw\" (UniqueName: \"kubernetes.io/projected/6b6fb240-4df1-45d9-8b43-319639d4087f-kube-api-access-5j5tw\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:58:55.327046 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.326815    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6fb240-4df1-45d9-8b43-319639d4087f-credential-socket\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:58:55.327046 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:55.326826    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6fb240-4df1-45d9-8b43-319639d4087f-istiod-ca-cert\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:58:56.089590 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:56.089508    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"
Jul 22 02:58:56.089590 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:56.089554    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv" event={"ID":"6b6fb240-4df1-45d9-8b43-319639d4087f","Type":"ContainerDied","Data":"4da46519d2d3eb50fb6fc551a9e8498867b833ce30fab8c293bd292484169b0c"}
Jul 22 02:58:56.090059 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:56.089607    2565 scope.go:117] "RemoveContainer" containerID="01b1933230b2ad9af453022cc64b6b643c9793b8a3f5c8bfc5d8b88cb88318dc"
Jul 22 02:58:56.110240 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:56.110203    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"]
Jul 22 02:58:56.118981 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:56.118956    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-187dc9-openshift-default-545c6d8b5-zhchv"]
Jul 22 02:58:57.726958 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:57.726923    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b6fb240-4df1-45d9-8b43-319639d4087f" path="/var/lib/kubelet/pods/6b6fb240-4df1-45d9-8b43-319639d4087f/volumes"
Jul 22 02:58:58.669464 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:58:58.669428    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:00.893267 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:00.893223    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"]
Jul 22 02:59:00.893878 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:00.893467    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5" podUID="e31d7313-7a93-425d-b295-b2d385a411c9" containerName="maas-api" containerID="cri-o://8cea0d43f53f283e3ccc0ceed65f45753b101e6a3b0e0479ec28ee45eab9a7a8" gracePeriod=30
Jul 22 02:59:00.921256 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:00.921232    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:01.120190 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:01.120157    2565 generic.go:356] "Generic (PLEG): container finished" podID="e31d7313-7a93-425d-b295-b2d385a411c9" containerID="8cea0d43f53f283e3ccc0ceed65f45753b101e6a3b0e0479ec28ee45eab9a7a8" exitCode=0
Jul 22 02:59:01.120340 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:01.120218    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5" event={"ID":"e31d7313-7a93-425d-b295-b2d385a411c9","Type":"ContainerDied","Data":"8cea0d43f53f283e3ccc0ceed65f45753b101e6a3b0e0479ec28ee45eab9a7a8"}
Jul 22 02:59:01.142556 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:01.142533    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"
Jul 22 02:59:01.278768 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:01.278690    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e31d7313-7a93-425d-b295-b2d385a411c9-maas-api-tls\") pod \"e31d7313-7a93-425d-b295-b2d385a411c9\" (UID: \"e31d7313-7a93-425d-b295-b2d385a411c9\") "
Jul 22 02:59:01.278768 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:01.278734    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31d7313-7a93-425d-b295-b2d385a411c9-openshift-service-ca\") pod \"e31d7313-7a93-425d-b295-b2d385a411c9\" (UID: \"e31d7313-7a93-425d-b295-b2d385a411c9\") "
Jul 22 02:59:01.278768 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:01.278767    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w67m7\" (UniqueName: \"kubernetes.io/projected/e31d7313-7a93-425d-b295-b2d385a411c9-kube-api-access-w67m7\") pod \"e31d7313-7a93-425d-b295-b2d385a411c9\" (UID: \"e31d7313-7a93-425d-b295-b2d385a411c9\") "
Jul 22 02:59:01.279126 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:01.279103    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e31d7313-7a93-425d-b295-b2d385a411c9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e31d7313-7a93-425d-b295-b2d385a411c9" (UID: "e31d7313-7a93-425d-b295-b2d385a411c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:59:01.280744 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:01.280724    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e31d7313-7a93-425d-b295-b2d385a411c9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e31d7313-7a93-425d-b295-b2d385a411c9" (UID: "e31d7313-7a93-425d-b295-b2d385a411c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:59:01.280864 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:01.280796    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31d7313-7a93-425d-b295-b2d385a411c9-kube-api-access-w67m7" (OuterVolumeSpecName: "kube-api-access-w67m7") pod "e31d7313-7a93-425d-b295-b2d385a411c9" (UID: "e31d7313-7a93-425d-b295-b2d385a411c9"). InnerVolumeSpecName "kube-api-access-w67m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:59:01.379886 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:01.379831    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e31d7313-7a93-425d-b295-b2d385a411c9-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:59:01.379886 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:01.379879    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31d7313-7a93-425d-b295-b2d385a411c9-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:59:01.379886 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:01.379892    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w67m7\" (UniqueName: \"kubernetes.io/projected/e31d7313-7a93-425d-b295-b2d385a411c9-kube-api-access-w67m7\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:59:02.126474 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:02.126441    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"
Jul 22 02:59:02.126922 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:02.126435    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5" event={"ID":"e31d7313-7a93-425d-b295-b2d385a411c9","Type":"ContainerDied","Data":"d173a9dc3a6d20049a9f1db46590d36360133021a76e3a411a843746a9018244"}
Jul 22 02:59:02.126922 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:02.126584    2565 scope.go:117] "RemoveContainer" containerID="8cea0d43f53f283e3ccc0ceed65f45753b101e6a3b0e0479ec28ee45eab9a7a8"
Jul 22 02:59:02.145920 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:02.145895    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"]
Jul 22 02:59:02.150644 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:02.150624    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6ecc40-687f89b9b7-z2sh5"]
Jul 22 02:59:03.726202 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:03.726172    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e31d7313-7a93-425d-b295-b2d385a411c9" path="/var/lib/kubelet/pods/e31d7313-7a93-425d-b295-b2d385a411c9/volumes"
Jul 22 02:59:05.302395 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:05.302361    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ecc40-rknc8"]
Jul 22 02:59:05.307113 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:05.307084    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ecc40-rknc8"]
Jul 22 02:59:05.307248 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:05.307088    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:05.685084 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:05.685052    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:05.726161 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:05.726132    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="219c8f78-be9f-4086-baeb-976a1d7c3c64" path="/var/lib/kubelet/pods/219c8f78-be9f-4086-baeb-976a1d7c3c64/volumes"
Jul 22 02:59:11.801119 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:11.801085    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:11.869052 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:11.869011    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:12.077703 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:12.077569    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:12.312263 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:12.312222    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:12.676478 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:12.676441    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:12.992898 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:12.992799    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"]
Jul 22 02:59:12.993355 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:12.993341    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="219c8f78-be9f-4086-baeb-976a1d7c3c64" containerName="revoke-keys"
Jul 22 02:59:12.993427 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:12.993362    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="219c8f78-be9f-4086-baeb-976a1d7c3c64" containerName="revoke-keys"
Jul 22 02:59:12.993427 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:12.993380    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e31d7313-7a93-425d-b295-b2d385a411c9" containerName="maas-api"
Jul 22 02:59:12.993427 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:12.993388    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e31d7313-7a93-425d-b295-b2d385a411c9" containerName="maas-api"
Jul 22 02:59:12.993427 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:12.993403    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6fb240-4df1-45d9-8b43-319639d4087f" containerName="istio-proxy"
Jul 22 02:59:12.993427 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:12.993413    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6fb240-4df1-45d9-8b43-319639d4087f" containerName="istio-proxy"
Jul 22 02:59:12.993673 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:12.993430    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6fb240-4df1-45d9-8b43-319639d4087f" containerName="istio-proxy"
Jul 22 02:59:12.993673 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:12.993439    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6fb240-4df1-45d9-8b43-319639d4087f" containerName="istio-proxy"
Jul 22 02:59:12.993673 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:12.993525    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b6fb240-4df1-45d9-8b43-319639d4087f" containerName="istio-proxy"
Jul 22 02:59:12.993673 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:12.993537    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e31d7313-7a93-425d-b295-b2d385a411c9" containerName="maas-api"
Jul 22 02:59:12.993673 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:12.993547    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b6fb240-4df1-45d9-8b43-319639d4087f" containerName="istio-proxy"
Jul 22 02:59:12.993673 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:12.993561    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="219c8f78-be9f-4086-baeb-976a1d7c3c64" containerName="revoke-keys"
Jul 22 02:59:12.996438 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:12.996417    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:12.999223 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:12.999202    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-95ff11-openshift-default-dockercfg-5jbgx\""
Jul 22 02:59:13.009820 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.009800    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"]
Jul 22 02:59:13.026523 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.026500    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:13.080606 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.080577    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-envoy\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.080767 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.080618    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-workload-certs\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.080767 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.080677    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241ffd09-72d3-40b6-b749-91cc116b66f7-istiod-ca-cert\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.080767 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.080715    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-podinfo\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.080767 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.080745    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-data\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.080980 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.080771    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckbbj\" (UniqueName: \"kubernetes.io/projected/241ffd09-72d3-40b6-b749-91cc116b66f7-kube-api-access-ckbbj\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.080980 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.080817    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-workload-socket\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.080980 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.080904    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-token\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.080980 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.080949    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-credential-socket\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.124268 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.124235    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:13.182264 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.182231    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-credential-socket\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.182432 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.182311    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-envoy\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.182432 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.182352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-workload-certs\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.182432 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.182390    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241ffd09-72d3-40b6-b749-91cc116b66f7-istiod-ca-cert\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.182598 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.182435    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-podinfo\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.182598 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.182464    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-data\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.182598 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.182494    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckbbj\" (UniqueName: \"kubernetes.io/projected/241ffd09-72d3-40b6-b749-91cc116b66f7-kube-api-access-ckbbj\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.182598 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.182535    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-workload-socket\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.182598 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.182578    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-token\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.182879 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.182735    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-credential-socket\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.182879 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.182739    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-workload-certs\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.182879 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.182799    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-data\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.183116 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.183094    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-workload-socket\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.183230 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.183209    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241ffd09-72d3-40b6-b749-91cc116b66f7-istiod-ca-cert\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.185326 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.185297    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-envoy\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.185484 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.185462    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-podinfo\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.196380 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.196361    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckbbj\" (UniqueName: \"kubernetes.io/projected/241ffd09-72d3-40b6-b749-91cc116b66f7-kube-api-access-ckbbj\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.196467 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.196443    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-token\") pod \"e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") " pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.306760 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.306700    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:13.453177 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.453151    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"]
Jul 22 02:59:13.455122 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:59:13.455093    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241ffd09_72d3_40b6_b749_91cc116b66f7.slice/crio-0afdb172b727f2dcb554e2cb5fd2e72135eb02996438abf768727c215d70632b WatchSource:0}: Error finding container 0afdb172b727f2dcb554e2cb5fd2e72135eb02996438abf768727c215d70632b: Status 404 returned error can't find the container with id 0afdb172b727f2dcb554e2cb5fd2e72135eb02996438abf768727c215d70632b
Jul 22 02:59:13.457156 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.457126    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:59:13.457222 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.457194    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:59:13.457263 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:13.457225    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:59:14.186392 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:14.186349    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv" event={"ID":"241ffd09-72d3-40b6-b749-91cc116b66f7","Type":"ContainerStarted","Data":"c064d69821895802647190583e727e3da5f76ac92edb892a2123720b4526650f"}
Jul 22 02:59:14.186908 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:14.186416    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv" event={"ID":"241ffd09-72d3-40b6-b749-91cc116b66f7","Type":"ContainerStarted","Data":"0afdb172b727f2dcb554e2cb5fd2e72135eb02996438abf768727c215d70632b"}
Jul 22 02:59:14.219110 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:14.219067    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv" podStartSLOduration=2.219052528 podStartE2EDuration="2.219052528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:59:14.21799999 +0000 UTC m=+2305.066336563" watchObservedRunningTime="2026-07-22 02:59:14.219052528 +0000 UTC m=+2305.067389103"
Jul 22 02:59:14.307159 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:14.307132    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:14.311796 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:14.311775    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:15.191756 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:15.191725    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:15.192775 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:15.192751    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:16.247225 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:16.247185    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:16.332280 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:16.332246    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"]
Jul 22 02:59:16.335160 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:16.335142    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"
Jul 22 02:59:16.341373 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:16.341347    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-95ff11-serving-cert\""
Jul 22 02:59:16.351618 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:16.351585    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"]
Jul 22 02:59:16.359891 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:16.359859    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:16.460453 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:16.460419    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:16.512914 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:16.512823    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4e0104-cbe8-4690-ba12-a1402deb0842-openshift-service-ca\") pod \"maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh\" (UID: \"ab4e0104-cbe8-4690-ba12-a1402deb0842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"
Jul 22 02:59:16.513062 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:16.512910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab4e0104-cbe8-4690-ba12-a1402deb0842-maas-api-tls\") pod \"maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh\" (UID: \"ab4e0104-cbe8-4690-ba12-a1402deb0842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"
Jul 22 02:59:16.513062 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:16.512974    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgn54\" (UniqueName: \"kubernetes.io/projected/ab4e0104-cbe8-4690-ba12-a1402deb0842-kube-api-access-kgn54\") pod \"maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh\" (UID: \"ab4e0104-cbe8-4690-ba12-a1402deb0842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"
Jul 22 02:59:16.613685 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:16.613655    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4e0104-cbe8-4690-ba12-a1402deb0842-openshift-service-ca\") pod \"maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh\" (UID: \"ab4e0104-cbe8-4690-ba12-a1402deb0842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"
Jul 22 02:59:16.613884 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:16.613689    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab4e0104-cbe8-4690-ba12-a1402deb0842-maas-api-tls\") pod \"maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh\" (UID: \"ab4e0104-cbe8-4690-ba12-a1402deb0842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"
Jul 22 02:59:16.613884 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:16.613730    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgn54\" (UniqueName: \"kubernetes.io/projected/ab4e0104-cbe8-4690-ba12-a1402deb0842-kube-api-access-kgn54\") pod \"maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh\" (UID: \"ab4e0104-cbe8-4690-ba12-a1402deb0842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"
Jul 22 02:59:16.613884 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:59:16.613812    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-serving-cert: secret "maas-api-e2e-api-a-95ff11-serving-cert" not found
Jul 22 02:59:16.614072 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:59:16.613894    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab4e0104-cbe8-4690-ba12-a1402deb0842-maas-api-tls podName:ab4e0104-cbe8-4690-ba12-a1402deb0842 nodeName:}" failed. No retries permitted until 2026-07-22 02:59:17.113875213 +0000 UTC m=+2307.962211765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ab4e0104-cbe8-4690-ba12-a1402deb0842-maas-api-tls") pod "maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh" (UID: "ab4e0104-cbe8-4690-ba12-a1402deb0842") : secret "maas-api-e2e-api-a-95ff11-serving-cert" not found
Jul 22 02:59:16.614388 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:16.614366    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4e0104-cbe8-4690-ba12-a1402deb0842-openshift-service-ca\") pod \"maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh\" (UID: \"ab4e0104-cbe8-4690-ba12-a1402deb0842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"
Jul 22 02:59:16.627318 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:16.627292    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgn54\" (UniqueName: \"kubernetes.io/projected/ab4e0104-cbe8-4690-ba12-a1402deb0842-kube-api-access-kgn54\") pod \"maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh\" (UID: \"ab4e0104-cbe8-4690-ba12-a1402deb0842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"
Jul 22 02:59:16.852575 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:16.852546    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:17.069453 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:17.069419    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:17.118757 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:17.118684    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab4e0104-cbe8-4690-ba12-a1402deb0842-maas-api-tls\") pod \"maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh\" (UID: \"ab4e0104-cbe8-4690-ba12-a1402deb0842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"
Jul 22 02:59:17.121050 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:17.121021    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab4e0104-cbe8-4690-ba12-a1402deb0842-maas-api-tls\") pod \"maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh\" (UID: \"ab4e0104-cbe8-4690-ba12-a1402deb0842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"
Jul 22 02:59:17.164769 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:17.164741    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:17.246315 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:17.246285    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"
Jul 22 02:59:17.387879 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:17.387840    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"]
Jul 22 02:59:17.389826 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:59:17.389799    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e0104_cbe8_4690_ba12_a1402deb0842.slice/crio-037733bbf42e74cc9e8ac5ff3ecb9f402156ca7dbdec4c37b98b6e50861cf569 WatchSource:0}: Error finding container 037733bbf42e74cc9e8ac5ff3ecb9f402156ca7dbdec4c37b98b6e50861cf569: Status 404 returned error can't find the container with id 037733bbf42e74cc9e8ac5ff3ecb9f402156ca7dbdec4c37b98b6e50861cf569
Jul 22 02:59:17.391242 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:17.391224    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 02:59:18.206649 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:18.206616    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh" event={"ID":"ab4e0104-cbe8-4690-ba12-a1402deb0842","Type":"ContainerStarted","Data":"f90f0c47d682d5a46645e909ade62af68fbe0d94c7bbf37e2e19e5871bb2de81"}
Jul 22 02:59:18.206822 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:18.206670    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh" event={"ID":"ab4e0104-cbe8-4690-ba12-a1402deb0842","Type":"ContainerStarted","Data":"037733bbf42e74cc9e8ac5ff3ecb9f402156ca7dbdec4c37b98b6e50861cf569"}
Jul 22 02:59:18.206822 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:18.206691    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"
Jul 22 02:59:18.236319 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:18.236273    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh" podStartSLOduration=1.807529453 podStartE2EDuration="2.23626027s" totalImagesPullingTime="428.730817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:59:18.235134192 +0000 UTC m=+2309.083470766" watchObservedRunningTime="2026-07-22 02:59:18.23626027 +0000 UTC m=+2309.084596843"
Jul 22 02:59:20.657984 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:20.657927    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:21.124754 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:21.124726    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:21.466906 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:21.466806    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:21.792157 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:21.792070    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:24.221872 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:24.221821    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"
Jul 22 02:59:24.929993 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:24.929962    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:25.031173 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:25.031133    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"]
Jul 22 02:59:25.034569 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:25.034546    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"
Jul 22 02:59:25.036229 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:25.036208    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:25.040471 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:25.040451    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4b8e31-serving-cert\""
Jul 22 02:59:25.051947 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:25.051887    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"]
Jul 22 02:59:25.126830 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:25.126790    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:25.187840 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:25.187753    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8vxf\" (UniqueName: \"kubernetes.io/projected/050a09ef-3df9-4ec3-bfd0-228c8e58cf6c-kube-api-access-b8vxf\") pod \"maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v\" (UID: \"050a09ef-3df9-4ec3-bfd0-228c8e58cf6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"
Jul 22 02:59:25.187840 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:25.187806    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/050a09ef-3df9-4ec3-bfd0-228c8e58cf6c-openshift-service-ca\") pod \"maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v\" (UID: \"050a09ef-3df9-4ec3-bfd0-228c8e58cf6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"
Jul 22 02:59:25.188090 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:25.187922    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/050a09ef-3df9-4ec3-bfd0-228c8e58cf6c-maas-api-tls\") pod \"maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v\" (UID: \"050a09ef-3df9-4ec3-bfd0-228c8e58cf6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"
Jul 22 02:59:25.224987 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:25.224955    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:25.289426 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:25.289392    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8vxf\" (UniqueName: \"kubernetes.io/projected/050a09ef-3df9-4ec3-bfd0-228c8e58cf6c-kube-api-access-b8vxf\") pod \"maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v\" (UID: \"050a09ef-3df9-4ec3-bfd0-228c8e58cf6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"
Jul 22 02:59:25.289575 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:25.289435    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/050a09ef-3df9-4ec3-bfd0-228c8e58cf6c-openshift-service-ca\") pod \"maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v\" (UID: \"050a09ef-3df9-4ec3-bfd0-228c8e58cf6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"
Jul 22 02:59:25.289575 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:25.289533    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/050a09ef-3df9-4ec3-bfd0-228c8e58cf6c-maas-api-tls\") pod \"maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v\" (UID: \"050a09ef-3df9-4ec3-bfd0-228c8e58cf6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"
Jul 22 02:59:25.290175 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:25.290154    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/050a09ef-3df9-4ec3-bfd0-228c8e58cf6c-openshift-service-ca\") pod \"maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v\" (UID: \"050a09ef-3df9-4ec3-bfd0-228c8e58cf6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"
Jul 22 02:59:25.292044 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:25.292017    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/050a09ef-3df9-4ec3-bfd0-228c8e58cf6c-maas-api-tls\") pod \"maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v\" (UID: \"050a09ef-3df9-4ec3-bfd0-228c8e58cf6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"
Jul 22 02:59:25.297712 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:25.297681    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8vxf\" (UniqueName: \"kubernetes.io/projected/050a09ef-3df9-4ec3-bfd0-228c8e58cf6c-kube-api-access-b8vxf\") pod \"maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v\" (UID: \"050a09ef-3df9-4ec3-bfd0-228c8e58cf6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"
Jul 22 02:59:25.356581 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:25.356548    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"
Jul 22 02:59:25.533085 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:25.532987    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"]
Jul 22 02:59:25.545214 ip-10-0-140-200 kubenswrapper[2565]: W0722 02:59:25.545183    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050a09ef_3df9_4ec3_bfd0_228c8e58cf6c.slice/crio-623db2e672da5056e12c8f25454b6f1740c2b239a994d21dc85055cb3f9f7025 WatchSource:0}: Error finding container 623db2e672da5056e12c8f25454b6f1740c2b239a994d21dc85055cb3f9f7025: Status 404 returned error can't find the container with id 623db2e672da5056e12c8f25454b6f1740c2b239a994d21dc85055cb3f9f7025
Jul 22 02:59:25.669745 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:25.669712    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:25.768793 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:25.768719    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:26.258413 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:26.258364    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v" event={"ID":"050a09ef-3df9-4ec3-bfd0-228c8e58cf6c","Type":"ContainerStarted","Data":"2c77cf4d1f56a284f70eae7345726a3c048a4d4f35295ef2be0b061919c07b50"}
Jul 22 02:59:26.258969 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:26.258427    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v" event={"ID":"050a09ef-3df9-4ec3-bfd0-228c8e58cf6c","Type":"ContainerStarted","Data":"623db2e672da5056e12c8f25454b6f1740c2b239a994d21dc85055cb3f9f7025"}
Jul 22 02:59:26.258969 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:26.258507    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"
Jul 22 02:59:26.285213 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:26.285167    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v" podStartSLOduration=0.878022696 podStartE2EDuration="1.285150527s" totalImagesPullingTime="407.127831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:59:26.284713067 +0000 UTC m=+2317.133049652" watchObservedRunningTime="2026-07-22 02:59:26.285150527 +0000 UTC m=+2317.133487107"
Jul 22 02:59:29.511829 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:29.511798    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:30.193722 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:30.193686    2565 patch_prober.go:28] interesting pod/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 22 02:59:30.193941 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:30.193735    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv" podUID="241ffd09-72d3-40b6-b749-91cc116b66f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 22 02:59:30.277818 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:30.277790    2565 generic.go:356] "Generic (PLEG): container finished" podID="241ffd09-72d3-40b6-b749-91cc116b66f7" containerID="c064d69821895802647190583e727e3da5f76ac92edb892a2123720b4526650f" exitCode=0
Jul 22 02:59:30.277979 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:30.277879    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv" event={"ID":"241ffd09-72d3-40b6-b749-91cc116b66f7","Type":"ContainerDied","Data":"c064d69821895802647190583e727e3da5f76ac92edb892a2123720b4526650f"}
Jul 22 02:59:30.278242 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:30.278228    2565 scope.go:117] "RemoveContainer" containerID="c064d69821895802647190583e727e3da5f76ac92edb892a2123720b4526650f"
Jul 22 02:59:30.279244 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:30.279218    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:59:30.279325 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:30.279282    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:59:30.279325 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:30.279313    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 02:59:31.284217 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:31.284188    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv" event={"ID":"241ffd09-72d3-40b6-b749-91cc116b66f7","Type":"ContainerStarted","Data":"d3040b522dc474588a25eecde289735bcff9561dfd7bb112db76e190cb78b0ed"}
Jul 22 02:59:31.307760 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:31.307736    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:31.308019 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:31.307999    2565 patch_prober.go:28] interesting pod/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 22 02:59:31.308085 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:31.308068    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv" podUID="241ffd09-72d3-40b6-b749-91cc116b66f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 22 02:59:32.268698 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:32.268669    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"
Jul 22 02:59:32.307652 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:32.307621    2565 patch_prober.go:28] interesting pod/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 22 02:59:32.308054 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:32.307670    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv" podUID="241ffd09-72d3-40b6-b749-91cc116b66f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 22 02:59:33.308081 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:33.308048    2565 patch_prober.go:28] interesting pod/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 22 02:59:33.308468 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:33.308101    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv" podUID="241ffd09-72d3-40b6-b749-91cc116b66f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 22 02:59:34.307805 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:34.307772    2565 patch_prober.go:28] interesting pod/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 22 02:59:34.307994 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:34.307824    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv" podUID="241ffd09-72d3-40b6-b749-91cc116b66f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 22 02:59:35.311480 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:35.311448    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:35.311909 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:35.311684    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:35.312509 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:35.312494    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 02:59:38.004308 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:38.004279    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:38.106573 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:38.106524    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:47.839404 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:47.839321    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:47.926105 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:47.926072    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:53.103560 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.103524    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"]
Jul 22 02:59:53.103971 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.103717    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh" podUID="ab4e0104-cbe8-4690-ba12-a1402deb0842" containerName="maas-api" containerID="cri-o://f90f0c47d682d5a46645e909ade62af68fbe0d94c7bbf37e2e19e5871bb2de81" gracePeriod=30
Jul 22 02:59:53.120608 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.120586    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:53.356889 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.356816    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"
Jul 22 02:59:53.397958 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.397930    2565 generic.go:356] "Generic (PLEG): container finished" podID="ab4e0104-cbe8-4690-ba12-a1402deb0842" containerID="f90f0c47d682d5a46645e909ade62af68fbe0d94c7bbf37e2e19e5871bb2de81" exitCode=0
Jul 22 02:59:53.398104 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.397967    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"
Jul 22 02:59:53.398104 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.398018    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh" event={"ID":"ab4e0104-cbe8-4690-ba12-a1402deb0842","Type":"ContainerDied","Data":"f90f0c47d682d5a46645e909ade62af68fbe0d94c7bbf37e2e19e5871bb2de81"}
Jul 22 02:59:53.398104 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.398074    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh" event={"ID":"ab4e0104-cbe8-4690-ba12-a1402deb0842","Type":"ContainerDied","Data":"037733bbf42e74cc9e8ac5ff3ecb9f402156ca7dbdec4c37b98b6e50861cf569"}
Jul 22 02:59:53.398104 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.398102    2565 scope.go:117] "RemoveContainer" containerID="f90f0c47d682d5a46645e909ade62af68fbe0d94c7bbf37e2e19e5871bb2de81"
Jul 22 02:59:53.407945 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.407927    2565 scope.go:117] "RemoveContainer" containerID="f90f0c47d682d5a46645e909ade62af68fbe0d94c7bbf37e2e19e5871bb2de81"
Jul 22 02:59:53.408205 ip-10-0-140-200 kubenswrapper[2565]: E0722 02:59:53.408189    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f90f0c47d682d5a46645e909ade62af68fbe0d94c7bbf37e2e19e5871bb2de81\": container with ID starting with f90f0c47d682d5a46645e909ade62af68fbe0d94c7bbf37e2e19e5871bb2de81 not found: ID does not exist" containerID="f90f0c47d682d5a46645e909ade62af68fbe0d94c7bbf37e2e19e5871bb2de81"
Jul 22 02:59:53.408252 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.408213    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f90f0c47d682d5a46645e909ade62af68fbe0d94c7bbf37e2e19e5871bb2de81"} err="failed to get container status \"f90f0c47d682d5a46645e909ade62af68fbe0d94c7bbf37e2e19e5871bb2de81\": rpc error: code = NotFound desc = could not find container \"f90f0c47d682d5a46645e909ade62af68fbe0d94c7bbf37e2e19e5871bb2de81\": container with ID starting with f90f0c47d682d5a46645e909ade62af68fbe0d94c7bbf37e2e19e5871bb2de81 not found: ID does not exist"
Jul 22 02:59:53.439525 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.439500    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4e0104-cbe8-4690-ba12-a1402deb0842-openshift-service-ca\") pod \"ab4e0104-cbe8-4690-ba12-a1402deb0842\" (UID: \"ab4e0104-cbe8-4690-ba12-a1402deb0842\") "
Jul 22 02:59:53.439632 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.439556    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgn54\" (UniqueName: \"kubernetes.io/projected/ab4e0104-cbe8-4690-ba12-a1402deb0842-kube-api-access-kgn54\") pod \"ab4e0104-cbe8-4690-ba12-a1402deb0842\" (UID: \"ab4e0104-cbe8-4690-ba12-a1402deb0842\") "
Jul 22 02:59:53.439632 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.439614    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab4e0104-cbe8-4690-ba12-a1402deb0842-maas-api-tls\") pod \"ab4e0104-cbe8-4690-ba12-a1402deb0842\" (UID: \"ab4e0104-cbe8-4690-ba12-a1402deb0842\") "
Jul 22 02:59:53.439921 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.439894    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab4e0104-cbe8-4690-ba12-a1402deb0842-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ab4e0104-cbe8-4690-ba12-a1402deb0842" (UID: "ab4e0104-cbe8-4690-ba12-a1402deb0842"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:59:53.441643 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.441616    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4e0104-cbe8-4690-ba12-a1402deb0842-kube-api-access-kgn54" (OuterVolumeSpecName: "kube-api-access-kgn54") pod "ab4e0104-cbe8-4690-ba12-a1402deb0842" (UID: "ab4e0104-cbe8-4690-ba12-a1402deb0842"). InnerVolumeSpecName "kube-api-access-kgn54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:59:53.441728 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.441643    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab4e0104-cbe8-4690-ba12-a1402deb0842-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ab4e0104-cbe8-4690-ba12-a1402deb0842" (UID: "ab4e0104-cbe8-4690-ba12-a1402deb0842"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:59:53.540371 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.540345    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgn54\" (UniqueName: \"kubernetes.io/projected/ab4e0104-cbe8-4690-ba12-a1402deb0842-kube-api-access-kgn54\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:59:53.540371 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.540370    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab4e0104-cbe8-4690-ba12-a1402deb0842-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:59:53.540507 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.540386    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4e0104-cbe8-4690-ba12-a1402deb0842-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 02:59:53.725858 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.725813    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"]
Jul 22 02:59:53.726002 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:53.725899    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95ff11-8699798dd6-jqqjh"]
Jul 22 02:59:55.726421 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:55.726387    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab4e0104-cbe8-4690-ba12-a1402deb0842" path="/var/lib/kubelet/pods/ab4e0104-cbe8-4690-ba12-a1402deb0842/volumes"
Jul 22 02:59:57.792469 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:57.792442    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 02:59:58.175524 ip-10-0-140-200 kubenswrapper[2565]: I0722 02:59:58.175495    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:00.146334 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:00.146297    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744820-dvcgf"]
Jul 22 03:00:00.146759 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:00.146717    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab4e0104-cbe8-4690-ba12-a1402deb0842" containerName="maas-api"
Jul 22 03:00:00.146759 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:00.146730    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab4e0104-cbe8-4690-ba12-a1402deb0842" containerName="maas-api"
Jul 22 03:00:00.146834 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:00.146808    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab4e0104-cbe8-4690-ba12-a1402deb0842" containerName="maas-api"
Jul 22 03:00:00.149893 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:00.149875    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744820-dvcgf"
Jul 22 03:00:00.152703 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:00.152681    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vtlqd\""
Jul 22 03:00:00.157825 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:00.157804    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744820-dvcgf"]
Jul 22 03:00:00.161427 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:00.161406    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744820-dvcgf"
Jul 22 03:00:00.306890 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:00.306839    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744820-dvcgf"]
Jul 22 03:00:00.308433 ip-10-0-140-200 kubenswrapper[2565]: W0722 03:00:00.308405    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f7a59a_f6ab_46f0_9da7_8b5a5f66ef47.slice/crio-daeb157d964966b4b0b2080714871ed5257411f484dfd5b238b66e94169a0c48 WatchSource:0}: Error finding container daeb157d964966b4b0b2080714871ed5257411f484dfd5b238b66e94169a0c48: Status 404 returned error can't find the container with id daeb157d964966b4b0b2080714871ed5257411f484dfd5b238b66e94169a0c48
Jul 22 03:00:00.431424 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:00.431390    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744820-dvcgf" event={"ID":"64f7a59a-f6ab-46f0-9da7-8b5a5f66ef47","Type":"ContainerStarted","Data":"daeb157d964966b4b0b2080714871ed5257411f484dfd5b238b66e94169a0c48"}
Jul 22 03:00:01.437325 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:01.437233    2565 generic.go:356] "Generic (PLEG): container finished" podID="64f7a59a-f6ab-46f0-9da7-8b5a5f66ef47" containerID="49de17da37d1b3d60ec0d8d9df031b4f0b993f6caef8659fdc54ab7feb8dbd19" exitCode=0
Jul 22 03:00:01.437325 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:01.437276    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744820-dvcgf" event={"ID":"64f7a59a-f6ab-46f0-9da7-8b5a5f66ef47","Type":"ContainerDied","Data":"49de17da37d1b3d60ec0d8d9df031b4f0b993f6caef8659fdc54ab7feb8dbd19"}
Jul 22 03:00:02.573607 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:02.573584    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744820-dvcgf"
Jul 22 03:00:03.447439 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:03.447411    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744820-dvcgf"
Jul 22 03:00:03.447439 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:03.447415    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744820-dvcgf" event={"ID":"64f7a59a-f6ab-46f0-9da7-8b5a5f66ef47","Type":"ContainerDied","Data":"daeb157d964966b4b0b2080714871ed5257411f484dfd5b238b66e94169a0c48"}
Jul 22 03:00:03.447723 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:03.447457    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daeb157d964966b4b0b2080714871ed5257411f484dfd5b238b66e94169a0c48"
Jul 22 03:00:04.129374 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:04.129341    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:04.228689 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:04.228653    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:04.387780 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:04.387701    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"]
Jul 22 03:00:04.388043 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:04.387985    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv" podUID="241ffd09-72d3-40b6-b749-91cc116b66f7" containerName="istio-proxy" containerID="cri-o://d3040b522dc474588a25eecde289735bcff9561dfd7bb112db76e190cb78b0ed" gracePeriod=30
Jul 22 03:00:04.418022 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:04.417993    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:04.646358 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:04.646273    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:04.762238 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:04.762193    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b8e31-vdbrk"]
Jul 22 03:00:04.762588 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:04.762574    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f7a59a-f6ab-46f0-9da7-8b5a5f66ef47" containerName="cleanup"
Jul 22 03:00:04.762588 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:04.762588    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f7a59a-f6ab-46f0-9da7-8b5a5f66ef47" containerName="cleanup"
Jul 22 03:00:04.762723 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:04.762653    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="64f7a59a-f6ab-46f0-9da7-8b5a5f66ef47" containerName="cleanup"
Jul 22 03:00:04.765742 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:04.765719    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b8e31-vdbrk"
Jul 22 03:00:04.768330 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:04.768308    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vtlqd\""
Jul 22 03:00:04.773705 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:04.773648    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b8e31-vdbrk"]
Jul 22 03:00:04.841242 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:04.841205    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00265d1e-6d69-4f5a-b9f2-a05104efb817-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4b8e31-vdbrk\" (UID: \"00265d1e-6d69-4f5a-b9f2-a05104efb817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b8e31-vdbrk"
Jul 22 03:00:04.942548 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:04.942470    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00265d1e-6d69-4f5a-b9f2-a05104efb817-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4b8e31-vdbrk\" (UID: \"00265d1e-6d69-4f5a-b9f2-a05104efb817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b8e31-vdbrk"
Jul 22 03:00:04.943064 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:04.943049    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00265d1e-6d69-4f5a-b9f2-a05104efb817-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4b8e31-vdbrk\" (UID: \"00265d1e-6d69-4f5a-b9f2-a05104efb817\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b8e31-vdbrk"
Jul 22 03:00:05.078223 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:05.078183    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b8e31-vdbrk"
Jul 22 03:00:05.213780 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:05.213757    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b8e31-vdbrk"]
Jul 22 03:00:05.215373 ip-10-0-140-200 kubenswrapper[2565]: W0722 03:00:05.215348    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00265d1e_6d69_4f5a_b9f2_a05104efb817.slice/crio-afbd1acd7dc34299bcfd52de72b2ab37d4d254b553bb948361ffbc0ed3245a0c WatchSource:0}: Error finding container afbd1acd7dc34299bcfd52de72b2ab37d4d254b553bb948361ffbc0ed3245a0c: Status 404 returned error can't find the container with id afbd1acd7dc34299bcfd52de72b2ab37d4d254b553bb948361ffbc0ed3245a0c
Jul 22 03:00:05.313717 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:05.313676    2565 patch_prober.go:28] interesting pod/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 22 03:00:05.313885 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:05.313742    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv" podUID="241ffd09-72d3-40b6-b749-91cc116b66f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 22 03:00:05.458617 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:05.458586    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b8e31-vdbrk" event={"ID":"00265d1e-6d69-4f5a-b9f2-a05104efb817","Type":"ContainerStarted","Data":"afbd1acd7dc34299bcfd52de72b2ab37d4d254b553bb948361ffbc0ed3245a0c"}
Jul 22 03:00:06.464966 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:06.464928    2565 generic.go:356] "Generic (PLEG): container finished" podID="00265d1e-6d69-4f5a-b9f2-a05104efb817" containerID="42becd366569073d1838252addee0e931e9782394c5cce30b2905591290da847" exitCode=0
Jul 22 03:00:06.465332 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:06.465007    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b8e31-vdbrk" event={"ID":"00265d1e-6d69-4f5a-b9f2-a05104efb817","Type":"ContainerDied","Data":"42becd366569073d1838252addee0e931e9782394c5cce30b2905591290da847"}
Jul 22 03:00:07.600886 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:07.600865    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b8e31-vdbrk"
Jul 22 03:00:07.765082 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:07.764994    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00265d1e-6d69-4f5a-b9f2-a05104efb817-maas-api-service-ca\") pod \"00265d1e-6d69-4f5a-b9f2-a05104efb817\" (UID: \"00265d1e-6d69-4f5a-b9f2-a05104efb817\") "
Jul 22 03:00:07.765336 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:07.765314    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00265d1e-6d69-4f5a-b9f2-a05104efb817-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "00265d1e-6d69-4f5a-b9f2-a05104efb817" (UID: "00265d1e-6d69-4f5a-b9f2-a05104efb817"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 03:00:07.866558 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:07.866527    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00265d1e-6d69-4f5a-b9f2-a05104efb817-maas-api-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:00:08.475401 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:08.475374    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b8e31-vdbrk"
Jul 22 03:00:08.475586 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:08.475424    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b8e31-vdbrk" event={"ID":"00265d1e-6d69-4f5a-b9f2-a05104efb817","Type":"ContainerDied","Data":"afbd1acd7dc34299bcfd52de72b2ab37d4d254b553bb948361ffbc0ed3245a0c"}
Jul 22 03:00:08.475586 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:08.475469    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afbd1acd7dc34299bcfd52de72b2ab37d4d254b553bb948361ffbc0ed3245a0c"
Jul 22 03:00:09.643724 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.643694    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 03:00:09.782096 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.782021    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-token\") pod \"241ffd09-72d3-40b6-b749-91cc116b66f7\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") "
Jul 22 03:00:09.782096 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.782057    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-workload-socket\") pod \"241ffd09-72d3-40b6-b749-91cc116b66f7\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") "
Jul 22 03:00:09.782293 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.782113    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-credential-socket\") pod \"241ffd09-72d3-40b6-b749-91cc116b66f7\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") "
Jul 22 03:00:09.782293 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.782141    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-podinfo\") pod \"241ffd09-72d3-40b6-b749-91cc116b66f7\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") "
Jul 22 03:00:09.782293 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.782163    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-data\") pod \"241ffd09-72d3-40b6-b749-91cc116b66f7\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") "
Jul 22 03:00:09.782293 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.782190    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-workload-certs\") pod \"241ffd09-72d3-40b6-b749-91cc116b66f7\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") "
Jul 22 03:00:09.782293 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.782222    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241ffd09-72d3-40b6-b749-91cc116b66f7-istiod-ca-cert\") pod \"241ffd09-72d3-40b6-b749-91cc116b66f7\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") "
Jul 22 03:00:09.782293 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.782265    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-envoy\") pod \"241ffd09-72d3-40b6-b749-91cc116b66f7\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") "
Jul 22 03:00:09.782293 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.782291    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckbbj\" (UniqueName: \"kubernetes.io/projected/241ffd09-72d3-40b6-b749-91cc116b66f7-kube-api-access-ckbbj\") pod \"241ffd09-72d3-40b6-b749-91cc116b66f7\" (UID: \"241ffd09-72d3-40b6-b749-91cc116b66f7\") "
Jul 22 03:00:09.782622 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.782448    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "241ffd09-72d3-40b6-b749-91cc116b66f7" (UID: "241ffd09-72d3-40b6-b749-91cc116b66f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:00:09.782622 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.782506    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "241ffd09-72d3-40b6-b749-91cc116b66f7" (UID: "241ffd09-72d3-40b6-b749-91cc116b66f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:00:09.782722 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.782619    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-workload-socket\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:00:09.782722 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.782636    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-credential-socket\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:00:09.782922 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.782897    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "241ffd09-72d3-40b6-b749-91cc116b66f7" (UID: "241ffd09-72d3-40b6-b749-91cc116b66f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:00:09.783026 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.782968    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/241ffd09-72d3-40b6-b749-91cc116b66f7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "241ffd09-72d3-40b6-b749-91cc116b66f7" (UID: "241ffd09-72d3-40b6-b749-91cc116b66f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 03:00:09.783106 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.783084    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-data" (OuterVolumeSpecName: "istio-data") pod "241ffd09-72d3-40b6-b749-91cc116b66f7" (UID: "241ffd09-72d3-40b6-b749-91cc116b66f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:00:09.784539 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.784511    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "241ffd09-72d3-40b6-b749-91cc116b66f7" (UID: "241ffd09-72d3-40b6-b749-91cc116b66f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:00:09.784655 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.784618    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-token" (OuterVolumeSpecName: "istio-token") pod "241ffd09-72d3-40b6-b749-91cc116b66f7" (UID: "241ffd09-72d3-40b6-b749-91cc116b66f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:00:09.784655 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.784615    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "241ffd09-72d3-40b6-b749-91cc116b66f7" (UID: "241ffd09-72d3-40b6-b749-91cc116b66f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 22 03:00:09.784784 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.784768    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241ffd09-72d3-40b6-b749-91cc116b66f7-kube-api-access-ckbbj" (OuterVolumeSpecName: "kube-api-access-ckbbj") pod "241ffd09-72d3-40b6-b749-91cc116b66f7" (UID: "241ffd09-72d3-40b6-b749-91cc116b66f7"). InnerVolumeSpecName "kube-api-access-ckbbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:00:09.883968 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.883931    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-podinfo\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:00:09.883968 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.883967    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-data\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:00:09.883968 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.883978    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-workload-certs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:00:09.884180 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.883988    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241ffd09-72d3-40b6-b749-91cc116b66f7-istiod-ca-cert\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:00:09.884180 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.883995    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-envoy\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:00:09.884180 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.884003    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckbbj\" (UniqueName: \"kubernetes.io/projected/241ffd09-72d3-40b6-b749-91cc116b66f7-kube-api-access-ckbbj\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:00:09.884180 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:09.884011    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241ffd09-72d3-40b6-b749-91cc116b66f7-istio-token\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:00:10.486178 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:10.486145    2565 generic.go:356] "Generic (PLEG): container finished" podID="241ffd09-72d3-40b6-b749-91cc116b66f7" containerID="d3040b522dc474588a25eecde289735bcff9561dfd7bb112db76e190cb78b0ed" exitCode=0
Jul 22 03:00:10.486380 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:10.486194    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"
Jul 22 03:00:10.486380 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:10.486233    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv" event={"ID":"241ffd09-72d3-40b6-b749-91cc116b66f7","Type":"ContainerDied","Data":"d3040b522dc474588a25eecde289735bcff9561dfd7bb112db76e190cb78b0ed"}
Jul 22 03:00:10.486380 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:10.486287    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv" event={"ID":"241ffd09-72d3-40b6-b749-91cc116b66f7","Type":"ContainerDied","Data":"0afdb172b727f2dcb554e2cb5fd2e72135eb02996438abf768727c215d70632b"}
Jul 22 03:00:10.486380 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:10.486310    2565 scope.go:117] "RemoveContainer" containerID="d3040b522dc474588a25eecde289735bcff9561dfd7bb112db76e190cb78b0ed"
Jul 22 03:00:10.495714 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:10.495696    2565 scope.go:117] "RemoveContainer" containerID="c064d69821895802647190583e727e3da5f76ac92edb892a2123720b4526650f"
Jul 22 03:00:10.503158 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:10.503139    2565 scope.go:117] "RemoveContainer" containerID="d3040b522dc474588a25eecde289735bcff9561dfd7bb112db76e190cb78b0ed"
Jul 22 03:00:10.503395 ip-10-0-140-200 kubenswrapper[2565]: E0722 03:00:10.503375    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3040b522dc474588a25eecde289735bcff9561dfd7bb112db76e190cb78b0ed\": container with ID starting with d3040b522dc474588a25eecde289735bcff9561dfd7bb112db76e190cb78b0ed not found: ID does not exist" containerID="d3040b522dc474588a25eecde289735bcff9561dfd7bb112db76e190cb78b0ed"
Jul 22 03:00:10.503442 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:10.503402    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3040b522dc474588a25eecde289735bcff9561dfd7bb112db76e190cb78b0ed"} err="failed to get container status \"d3040b522dc474588a25eecde289735bcff9561dfd7bb112db76e190cb78b0ed\": rpc error: code = NotFound desc = could not find container \"d3040b522dc474588a25eecde289735bcff9561dfd7bb112db76e190cb78b0ed\": container with ID starting with d3040b522dc474588a25eecde289735bcff9561dfd7bb112db76e190cb78b0ed not found: ID does not exist"
Jul 22 03:00:10.503442 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:10.503419    2565 scope.go:117] "RemoveContainer" containerID="c064d69821895802647190583e727e3da5f76ac92edb892a2123720b4526650f"
Jul 22 03:00:10.503612 ip-10-0-140-200 kubenswrapper[2565]: E0722 03:00:10.503598    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c064d69821895802647190583e727e3da5f76ac92edb892a2123720b4526650f\": container with ID starting with c064d69821895802647190583e727e3da5f76ac92edb892a2123720b4526650f not found: ID does not exist" containerID="c064d69821895802647190583e727e3da5f76ac92edb892a2123720b4526650f"
Jul 22 03:00:10.503657 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:10.503615    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c064d69821895802647190583e727e3da5f76ac92edb892a2123720b4526650f"} err="failed to get container status \"c064d69821895802647190583e727e3da5f76ac92edb892a2123720b4526650f\": rpc error: code = NotFound desc = could not find container \"c064d69821895802647190583e727e3da5f76ac92edb892a2123720b4526650f\": container with ID starting with c064d69821895802647190583e727e3da5f76ac92edb892a2123720b4526650f not found: ID does not exist"
Jul 22 03:00:10.508593 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:10.508570    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"]
Jul 22 03:00:10.510291 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:10.510273    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-95ff11-openshift-default-7b44f7d66c-82zzv"]
Jul 22 03:00:11.726255 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:11.726221    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="241ffd09-72d3-40b6-b749-91cc116b66f7" path="/var/lib/kubelet/pods/241ffd09-72d3-40b6-b749-91cc116b66f7/volumes"
Jul 22 03:00:20.217837 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.217800    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"]
Jul 22 03:00:20.218675 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.218644    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v" podUID="050a09ef-3df9-4ec3-bfd0-228c8e58cf6c" containerName="maas-api" containerID="cri-o://2c77cf4d1f56a284f70eae7345726a3c048a4d4f35295ef2be0b061919c07b50" gracePeriod=30
Jul 22 03:00:20.256080 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.256052    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:20.471726 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.471673    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"
Jul 22 03:00:20.536016 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.535988    2565 generic.go:356] "Generic (PLEG): container finished" podID="050a09ef-3df9-4ec3-bfd0-228c8e58cf6c" containerID="2c77cf4d1f56a284f70eae7345726a3c048a4d4f35295ef2be0b061919c07b50" exitCode=0
Jul 22 03:00:20.536158 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.536022    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"
Jul 22 03:00:20.536158 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.536060    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v" event={"ID":"050a09ef-3df9-4ec3-bfd0-228c8e58cf6c","Type":"ContainerDied","Data":"2c77cf4d1f56a284f70eae7345726a3c048a4d4f35295ef2be0b061919c07b50"}
Jul 22 03:00:20.536158 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.536110    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v" event={"ID":"050a09ef-3df9-4ec3-bfd0-228c8e58cf6c","Type":"ContainerDied","Data":"623db2e672da5056e12c8f25454b6f1740c2b239a994d21dc85055cb3f9f7025"}
Jul 22 03:00:20.536158 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.536133    2565 scope.go:117] "RemoveContainer" containerID="2c77cf4d1f56a284f70eae7345726a3c048a4d4f35295ef2be0b061919c07b50"
Jul 22 03:00:20.544317 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.544301    2565 scope.go:117] "RemoveContainer" containerID="2c77cf4d1f56a284f70eae7345726a3c048a4d4f35295ef2be0b061919c07b50"
Jul 22 03:00:20.544559 ip-10-0-140-200 kubenswrapper[2565]: E0722 03:00:20.544542    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c77cf4d1f56a284f70eae7345726a3c048a4d4f35295ef2be0b061919c07b50\": container with ID starting with 2c77cf4d1f56a284f70eae7345726a3c048a4d4f35295ef2be0b061919c07b50 not found: ID does not exist" containerID="2c77cf4d1f56a284f70eae7345726a3c048a4d4f35295ef2be0b061919c07b50"
Jul 22 03:00:20.544606 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.544567    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c77cf4d1f56a284f70eae7345726a3c048a4d4f35295ef2be0b061919c07b50"} err="failed to get container status \"2c77cf4d1f56a284f70eae7345726a3c048a4d4f35295ef2be0b061919c07b50\": rpc error: code = NotFound desc = could not find container \"2c77cf4d1f56a284f70eae7345726a3c048a4d4f35295ef2be0b061919c07b50\": container with ID starting with 2c77cf4d1f56a284f70eae7345726a3c048a4d4f35295ef2be0b061919c07b50 not found: ID does not exist"
Jul 22 03:00:20.575023 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.575002    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/050a09ef-3df9-4ec3-bfd0-228c8e58cf6c-maas-api-tls\") pod \"050a09ef-3df9-4ec3-bfd0-228c8e58cf6c\" (UID: \"050a09ef-3df9-4ec3-bfd0-228c8e58cf6c\") "
Jul 22 03:00:20.575112 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.575088    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8vxf\" (UniqueName: \"kubernetes.io/projected/050a09ef-3df9-4ec3-bfd0-228c8e58cf6c-kube-api-access-b8vxf\") pod \"050a09ef-3df9-4ec3-bfd0-228c8e58cf6c\" (UID: \"050a09ef-3df9-4ec3-bfd0-228c8e58cf6c\") "
Jul 22 03:00:20.575152 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.575124    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/050a09ef-3df9-4ec3-bfd0-228c8e58cf6c-openshift-service-ca\") pod \"050a09ef-3df9-4ec3-bfd0-228c8e58cf6c\" (UID: \"050a09ef-3df9-4ec3-bfd0-228c8e58cf6c\") "
Jul 22 03:00:20.575505 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.575481    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050a09ef-3df9-4ec3-bfd0-228c8e58cf6c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "050a09ef-3df9-4ec3-bfd0-228c8e58cf6c" (UID: "050a09ef-3df9-4ec3-bfd0-228c8e58cf6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 03:00:20.577085 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.577062    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/050a09ef-3df9-4ec3-bfd0-228c8e58cf6c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "050a09ef-3df9-4ec3-bfd0-228c8e58cf6c" (UID: "050a09ef-3df9-4ec3-bfd0-228c8e58cf6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 03:00:20.577085 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.577061    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050a09ef-3df9-4ec3-bfd0-228c8e58cf6c-kube-api-access-b8vxf" (OuterVolumeSpecName: "kube-api-access-b8vxf") pod "050a09ef-3df9-4ec3-bfd0-228c8e58cf6c" (UID: "050a09ef-3df9-4ec3-bfd0-228c8e58cf6c"). InnerVolumeSpecName "kube-api-access-b8vxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:00:20.675650 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.675615    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8vxf\" (UniqueName: \"kubernetes.io/projected/050a09ef-3df9-4ec3-bfd0-228c8e58cf6c-kube-api-access-b8vxf\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:00:20.675650 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.675644    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/050a09ef-3df9-4ec3-bfd0-228c8e58cf6c-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:00:20.675650 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.675654    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/050a09ef-3df9-4ec3-bfd0-228c8e58cf6c-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:00:20.857560 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.857532    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"]
Jul 22 03:00:20.865054 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:20.865027    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4b8e31-688b57b9f6-wgn6v"]
Jul 22 03:00:21.725661 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:21.725625    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="050a09ef-3df9-4ec3-bfd0-228c8e58cf6c" path="/var/lib/kubelet/pods/050a09ef-3df9-4ec3-bfd0-228c8e58cf6c/volumes"
Jul 22 03:00:24.849043 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:24.849008    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b8e31-vdbrk"]
Jul 22 03:00:24.852306 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:24.852279    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b8e31-vdbrk"]
Jul 22 03:00:24.861360 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:24.861337    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:25.241812 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:25.241779    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:25.725239 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:25.725209    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00265d1e-6d69-4f5a-b9f2-a05104efb817" path="/var/lib/kubelet/pods/00265d1e-6d69-4f5a-b9f2-a05104efb817/volumes"
Jul 22 03:00:32.251001 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.250967    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:32.360241 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.360208    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:32.410995 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.410962    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"]
Jul 22 03:00:32.411338 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.411325    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="241ffd09-72d3-40b6-b749-91cc116b66f7" containerName="istio-proxy"
Jul 22 03:00:32.411338 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.411339    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="241ffd09-72d3-40b6-b749-91cc116b66f7" containerName="istio-proxy"
Jul 22 03:00:32.411450 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.411356    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="050a09ef-3df9-4ec3-bfd0-228c8e58cf6c" containerName="maas-api"
Jul 22 03:00:32.411450 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.411362    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="050a09ef-3df9-4ec3-bfd0-228c8e58cf6c" containerName="maas-api"
Jul 22 03:00:32.411450 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.411372    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00265d1e-6d69-4f5a-b9f2-a05104efb817" containerName="revoke-keys"
Jul 22 03:00:32.411450 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.411377    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="00265d1e-6d69-4f5a-b9f2-a05104efb817" containerName="revoke-keys"
Jul 22 03:00:32.411450 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.411388    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="241ffd09-72d3-40b6-b749-91cc116b66f7" containerName="istio-proxy"
Jul 22 03:00:32.411450 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.411396    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="241ffd09-72d3-40b6-b749-91cc116b66f7" containerName="istio-proxy"
Jul 22 03:00:32.411646 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.411455    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="050a09ef-3df9-4ec3-bfd0-228c8e58cf6c" containerName="maas-api"
Jul 22 03:00:32.411646 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.411465    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="241ffd09-72d3-40b6-b749-91cc116b66f7" containerName="istio-proxy"
Jul 22 03:00:32.411646 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.411474    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="00265d1e-6d69-4f5a-b9f2-a05104efb817" containerName="revoke-keys"
Jul 22 03:00:32.411646 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.411484    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="241ffd09-72d3-40b6-b749-91cc116b66f7" containerName="istio-proxy"
Jul 22 03:00:32.414603 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.414579    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.419985 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.419965    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-d8076c-openshift-default-dockercfg-q9brq\""
Jul 22 03:00:32.437447 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.437420    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"]
Jul 22 03:00:32.468353 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.468318    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:32.483584 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.483556    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-token\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.483680 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.483589    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktbqw\" (UniqueName: \"kubernetes.io/projected/a225421d-45bf-4dcb-b18a-877fef022eeb-kube-api-access-ktbqw\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.483680 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.483614    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-credential-socket\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.483793 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.483690    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-podinfo\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.483793 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.483744    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-envoy\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.483793 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.483763    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a225421d-45bf-4dcb-b18a-877fef022eeb-istiod-ca-cert\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.483793 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.483783    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-data\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.484023 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.483840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-workload-socket\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.484023 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.483899    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-workload-certs\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.585028 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.584955    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-data\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.585028 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.584999    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-workload-socket\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.585224 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.585119    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-workload-certs\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.585224 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.585171    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-token\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.585224 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.585198    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktbqw\" (UniqueName: \"kubernetes.io/projected/a225421d-45bf-4dcb-b18a-877fef022eeb-kube-api-access-ktbqw\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.585380 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.585230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-credential-socket\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.585380 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.585280    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-podinfo\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.585380 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.585332    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-envoy\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.585380 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.585360    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a225421d-45bf-4dcb-b18a-877fef022eeb-istiod-ca-cert\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.585610 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.585384    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-data\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.585610 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.585426    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-workload-socket\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.585610 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.585512    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-workload-certs\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.585730 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.585701    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-credential-socket\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.586158 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.586130    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a225421d-45bf-4dcb-b18a-877fef022eeb-istiod-ca-cert\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.587661 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.587638    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-envoy\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.587838 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.587824    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-podinfo\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.594513 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.594484    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-token\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.595080 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.595058    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktbqw\" (UniqueName: \"kubernetes.io/projected/a225421d-45bf-4dcb-b18a-877fef022eeb-kube-api-access-ktbqw\") pod \"e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") " pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.727257 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.727215    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:32.749388 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.749357    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:32.866135 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.866110    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"]
Jul 22 03:00:32.868348 ip-10-0-140-200 kubenswrapper[2565]: W0722 03:00:32.868322    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda225421d_45bf_4dcb_b18a_877fef022eeb.slice/crio-f1f6ad8b693e9d374d3907313ab0aa7cdfb7b43a841395f79439ed3368c688d6 WatchSource:0}: Error finding container f1f6ad8b693e9d374d3907313ab0aa7cdfb7b43a841395f79439ed3368c688d6: Status 404 returned error can't find the container with id f1f6ad8b693e9d374d3907313ab0aa7cdfb7b43a841395f79439ed3368c688d6
Jul 22 03:00:32.870449 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.870413    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 03:00:32.870524 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.870481    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 03:00:32.870524 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:32.870512    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 03:00:33.600121 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:33.600088    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf" event={"ID":"a225421d-45bf-4dcb-b18a-877fef022eeb","Type":"ContainerStarted","Data":"e2bede040c9f889092783ca8d4217b6fd116b70aec7a49e0bd196aa77e3b7474"}
Jul 22 03:00:33.600547 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:33.600150    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf" event={"ID":"a225421d-45bf-4dcb-b18a-877fef022eeb","Type":"ContainerStarted","Data":"f1f6ad8b693e9d374d3907313ab0aa7cdfb7b43a841395f79439ed3368c688d6"}
Jul 22 03:00:33.623187 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:33.623134    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf" podStartSLOduration=1.623115474 podStartE2EDuration="1.623115474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:00:33.622779917 +0000 UTC m=+2384.471116494" watchObservedRunningTime="2026-07-22 03:00:33.623115474 +0000 UTC m=+2384.471452049"
Jul 22 03:00:33.727469 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:33.727438    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:33.735545 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:33.735520    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:34.604403 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:34.604371    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:34.605200 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:34.605182    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:35.563481 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:35.563450    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:35.663890 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:35.663833    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:35.772778 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:35.772752    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"]
Jul 22 03:00:35.776368 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:35.776350    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"
Jul 22 03:00:35.778762 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:35.778738    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-d8076c-serving-cert\""
Jul 22 03:00:35.787529 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:35.787504    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"]
Jul 22 03:00:35.818805 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:35.818749    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:35.914011 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:35.913982    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/801fcedf-8abe-4e56-be26-4d6adb7a8e28-maas-api-tls\") pod \"maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75\" (UID: \"801fcedf-8abe-4e56-be26-4d6adb7a8e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"
Jul 22 03:00:35.914132 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:35.914030    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/801fcedf-8abe-4e56-be26-4d6adb7a8e28-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75\" (UID: \"801fcedf-8abe-4e56-be26-4d6adb7a8e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"
Jul 22 03:00:35.914132 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:35.914078    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxvtg\" (UniqueName: \"kubernetes.io/projected/801fcedf-8abe-4e56-be26-4d6adb7a8e28-kube-api-access-kxvtg\") pod \"maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75\" (UID: \"801fcedf-8abe-4e56-be26-4d6adb7a8e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"
Jul 22 03:00:35.921911 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:35.921890    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:36.015159 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:36.015130    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/801fcedf-8abe-4e56-be26-4d6adb7a8e28-maas-api-tls\") pod \"maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75\" (UID: \"801fcedf-8abe-4e56-be26-4d6adb7a8e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"
Jul 22 03:00:36.015340 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:36.015182    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/801fcedf-8abe-4e56-be26-4d6adb7a8e28-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75\" (UID: \"801fcedf-8abe-4e56-be26-4d6adb7a8e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"
Jul 22 03:00:36.015340 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:36.015208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxvtg\" (UniqueName: \"kubernetes.io/projected/801fcedf-8abe-4e56-be26-4d6adb7a8e28-kube-api-access-kxvtg\") pod \"maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75\" (UID: \"801fcedf-8abe-4e56-be26-4d6adb7a8e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"
Jul 22 03:00:36.015885 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:36.015839    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/801fcedf-8abe-4e56-be26-4d6adb7a8e28-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75\" (UID: \"801fcedf-8abe-4e56-be26-4d6adb7a8e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"
Jul 22 03:00:36.017532 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:36.017513    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/801fcedf-8abe-4e56-be26-4d6adb7a8e28-maas-api-tls\") pod \"maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75\" (UID: \"801fcedf-8abe-4e56-be26-4d6adb7a8e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"
Jul 22 03:00:36.028492 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:36.028467    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxvtg\" (UniqueName: \"kubernetes.io/projected/801fcedf-8abe-4e56-be26-4d6adb7a8e28-kube-api-access-kxvtg\") pod \"maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75\" (UID: \"801fcedf-8abe-4e56-be26-4d6adb7a8e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"
Jul 22 03:00:36.088383 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:36.088302    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"
Jul 22 03:00:36.267613 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:36.267574    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"]
Jul 22 03:00:36.268423 ip-10-0-140-200 kubenswrapper[2565]: W0722 03:00:36.268396    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801fcedf_8abe_4e56_be26_4d6adb7a8e28.slice/crio-a08d0a97c09ebb2275a984cea8ad62d01ef1e246c2505f1d549ce0be7509b4c5 WatchSource:0}: Error finding container a08d0a97c09ebb2275a984cea8ad62d01ef1e246c2505f1d549ce0be7509b4c5: Status 404 returned error can't find the container with id a08d0a97c09ebb2275a984cea8ad62d01ef1e246c2505f1d549ce0be7509b4c5
Jul 22 03:00:36.485674 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:36.485643    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:36.587237 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:36.587188    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:36.614356 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:36.614323    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75" event={"ID":"801fcedf-8abe-4e56-be26-4d6adb7a8e28","Type":"ContainerStarted","Data":"a08d0a97c09ebb2275a984cea8ad62d01ef1e246c2505f1d549ce0be7509b4c5"}
Jul 22 03:00:37.381822 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:37.381788    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:37.484804 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:37.484763    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:37.621253 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:37.621215    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75" event={"ID":"801fcedf-8abe-4e56-be26-4d6adb7a8e28","Type":"ContainerStarted","Data":"f273a0934ee95ceaf6197fddfc72735623e8de1c87e07f624bfdbcf2c682ffb0"}
Jul 22 03:00:37.621428 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:37.621394    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"
Jul 22 03:00:37.644069 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:37.643975    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75" podStartSLOduration=2.142185384 podStartE2EDuration="2.643960485s" totalImagesPullingTime="501.775101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:00:37.642161186 +0000 UTC m=+2388.490497760" watchObservedRunningTime="2026-07-22 03:00:37.643960485 +0000 UTC m=+2388.492297059"
Jul 22 03:00:38.207695 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:38.207661    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:38.307344 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:38.307315    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:40.113298 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:40.113263    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:40.443726 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:40.443631    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:40.765990 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:40.765907    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:41.076379 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:41.076301    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:43.372152 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:43.372111    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:43.476466 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:43.476430    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:43.571789 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:43.571758    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"]
Jul 22 03:00:43.575275 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:43.575253    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"
Jul 22 03:00:43.577675 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:43.577654    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-cf82b3-serving-cert\""
Jul 22 03:00:43.588022 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:43.588001    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"]
Jul 22 03:00:43.612487 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:43.612464    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:43.631784 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:43.631733    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"
Jul 22 03:00:43.690469 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:43.690436    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58bb7202-f9fb-41a2-adfe-4d007f52e3fb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl\" (UID: \"58bb7202-f9fb-41a2-adfe-4d007f52e3fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"
Jul 22 03:00:43.690616 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:43.690510    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xv76\" (UniqueName: \"kubernetes.io/projected/58bb7202-f9fb-41a2-adfe-4d007f52e3fb-kube-api-access-5xv76\") pod \"maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl\" (UID: \"58bb7202-f9fb-41a2-adfe-4d007f52e3fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"
Jul 22 03:00:43.690666 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:43.690637    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58bb7202-f9fb-41a2-adfe-4d007f52e3fb-maas-api-tls\") pod \"maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl\" (UID: \"58bb7202-f9fb-41a2-adfe-4d007f52e3fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"
Jul 22 03:00:43.791950 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:43.791916    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xv76\" (UniqueName: \"kubernetes.io/projected/58bb7202-f9fb-41a2-adfe-4d007f52e3fb-kube-api-access-5xv76\") pod \"maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl\" (UID: \"58bb7202-f9fb-41a2-adfe-4d007f52e3fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"
Jul 22 03:00:43.792139 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:43.792039    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58bb7202-f9fb-41a2-adfe-4d007f52e3fb-maas-api-tls\") pod \"maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl\" (UID: \"58bb7202-f9fb-41a2-adfe-4d007f52e3fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"
Jul 22 03:00:43.792493 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:43.792468    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58bb7202-f9fb-41a2-adfe-4d007f52e3fb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl\" (UID: \"58bb7202-f9fb-41a2-adfe-4d007f52e3fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"
Jul 22 03:00:43.793185 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:43.793162    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58bb7202-f9fb-41a2-adfe-4d007f52e3fb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl\" (UID: \"58bb7202-f9fb-41a2-adfe-4d007f52e3fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"
Jul 22 03:00:43.795158 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:43.795139    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58bb7202-f9fb-41a2-adfe-4d007f52e3fb-maas-api-tls\") pod \"maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl\" (UID: \"58bb7202-f9fb-41a2-adfe-4d007f52e3fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"
Jul 22 03:00:43.800325 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:43.800301    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xv76\" (UniqueName: \"kubernetes.io/projected/58bb7202-f9fb-41a2-adfe-4d007f52e3fb-kube-api-access-5xv76\") pod \"maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl\" (UID: \"58bb7202-f9fb-41a2-adfe-4d007f52e3fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"
Jul 22 03:00:43.833104 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:43.833082    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:43.888197 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:43.888136    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"
Jul 22 03:00:44.043254 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:44.043229    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"]
Jul 22 03:00:44.043910 ip-10-0-140-200 kubenswrapper[2565]: W0722 03:00:44.043838    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58bb7202_f9fb_41a2_adfe_4d007f52e3fb.slice/crio-4d907f463937c2213a9c53cc4ff201481daf8466e3a4c4b9055f2d9172b91b9a WatchSource:0}: Error finding container 4d907f463937c2213a9c53cc4ff201481daf8466e3a4c4b9055f2d9172b91b9a: Status 404 returned error can't find the container with id 4d907f463937c2213a9c53cc4ff201481daf8466e3a4c4b9055f2d9172b91b9a
Jul 22 03:00:44.235734 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:44.235646    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:44.336371 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:44.336336    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:44.659910 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:44.659869    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl" event={"ID":"58bb7202-f9fb-41a2-adfe-4d007f52e3fb","Type":"ContainerStarted","Data":"c04ed884d5e97fc69c6200035141e522eb7f185bd12438b30336771d00e7c438"}
Jul 22 03:00:44.660356 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:44.659928    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl" event={"ID":"58bb7202-f9fb-41a2-adfe-4d007f52e3fb","Type":"ContainerStarted","Data":"4d907f463937c2213a9c53cc4ff201481daf8466e3a4c4b9055f2d9172b91b9a"}
Jul 22 03:00:44.660356 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:44.660017    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"
Jul 22 03:00:44.678370 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:44.678322    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl" podStartSLOduration=1.239364073 podStartE2EDuration="1.678308057s" totalImagesPullingTime="438.943984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:00:44.677078324 +0000 UTC m=+2395.525414912" watchObservedRunningTime="2026-07-22 03:00:44.678308057 +0000 UTC m=+2395.526644635"
Jul 22 03:00:47.862558 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:47.862523    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:50.669801 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:50.669768    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"
Jul 22 03:00:53.707374 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:53.707341    2565 generic.go:356] "Generic (PLEG): container finished" podID="a225421d-45bf-4dcb-b18a-877fef022eeb" containerID="e2bede040c9f889092783ca8d4217b6fd116b70aec7a49e0bd196aa77e3b7474" exitCode=0
Jul 22 03:00:53.707773 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:53.707409    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf" event={"ID":"a225421d-45bf-4dcb-b18a-877fef022eeb","Type":"ContainerDied","Data":"e2bede040c9f889092783ca8d4217b6fd116b70aec7a49e0bd196aa77e3b7474"}
Jul 22 03:00:53.707831 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:53.707815    2565 scope.go:117] "RemoveContainer" containerID="e2bede040c9f889092783ca8d4217b6fd116b70aec7a49e0bd196aa77e3b7474"
Jul 22 03:00:53.708893 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:53.708843    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 03:00:53.708965 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:53.708936    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 03:00:53.709003 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:53.708966    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 22 03:00:54.714273 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:54.714234    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf" event={"ID":"a225421d-45bf-4dcb-b18a-877fef022eeb","Type":"ContainerStarted","Data":"ba1eaec9b3620d534811919c01b4f06c8c228bfa8b72ebfa855bdf9b3a1040c2"}
Jul 22 03:00:54.727998 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:54.727950    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:54.728194 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:54.728025    2565 patch_prober.go:28] interesting pod/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 22 03:00:54.728194 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:54.728114    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf" podUID="a225421d-45bf-4dcb-b18a-877fef022eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 22 03:00:55.729270 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:55.728820    2565 patch_prober.go:28] interesting pod/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 22 03:00:55.729270 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:55.728906    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf" podUID="a225421d-45bf-4dcb-b18a-877fef022eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 22 03:00:56.729317 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:56.729282    2565 patch_prober.go:28] interesting pod/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 22 03:00:56.729890 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:56.729839    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf" podUID="a225421d-45bf-4dcb-b18a-877fef022eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 22 03:00:57.489405 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:57.489361    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:57.590774 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:57.590743    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:00:57.730560 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:57.730525    2565 patch_prober.go:28] interesting pod/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 22 03:00:57.731124 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:57.731093    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf" podUID="a225421d-45bf-4dcb-b18a-877fef022eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 22 03:00:58.731210 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:58.731181    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:58.731625 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:58.731564    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:00:58.732269 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:00:58.732246    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:01:04.990352 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:04.990304    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:01:07.073252 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:07.073213    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:01:11.617588 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:11.617515    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:01:11.679778 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:11.679744    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:01:29.526227 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:29.526182    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:01:57.988604 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:57.988570    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"]
Jul 22 03:01:57.992079 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:57.992055    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:01:57.995367 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:57.995340    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cf82b3\"/\"kube-root-ca.crt\""
Jul 22 03:01:57.995644 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:57.995590    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cf82b3\"/\"openshift-service-ca.crt\""
Jul 22 03:01:57.996452 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:57.996432    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cf82b3\"/\"default-dockercfg-xzbgs\""
Jul 22 03:01:57.996569 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:57.996482    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cf82b3\"/\"auth-test-model-cf82b3-kserve-self-signed-certs\""
Jul 22 03:01:58.003449 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.003423    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"]
Jul 22 03:01:58.027757 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.027730    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:01:58.049354 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.049321    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bf5f637-1082-42f9-b0ba-dd0f3b274956-tls-certs\") pod \"auth-test-model-cf82b3-kserve-575c9774bc-4bmzs\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") " pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:01:58.049532 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.049431    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-dshm\") pod \"auth-test-model-cf82b3-kserve-575c9774bc-4bmzs\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") " pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:01:58.049532 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.049499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7lg6\" (UniqueName: \"kubernetes.io/projected/8bf5f637-1082-42f9-b0ba-dd0f3b274956-kube-api-access-r7lg6\") pod \"auth-test-model-cf82b3-kserve-575c9774bc-4bmzs\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") " pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:01:58.049532 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.049528    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-home\") pod \"auth-test-model-cf82b3-kserve-575c9774bc-4bmzs\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") " pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:01:58.049688 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.049611    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-model-cache\") pod \"auth-test-model-cf82b3-kserve-575c9774bc-4bmzs\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") " pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:01:58.049688 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.049672    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-tmp-dir\") pod \"auth-test-model-cf82b3-kserve-575c9774bc-4bmzs\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") " pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:01:58.150418 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.150382    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7lg6\" (UniqueName: \"kubernetes.io/projected/8bf5f637-1082-42f9-b0ba-dd0f3b274956-kube-api-access-r7lg6\") pod \"auth-test-model-cf82b3-kserve-575c9774bc-4bmzs\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") " pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:01:58.150610 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.150427    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-home\") pod \"auth-test-model-cf82b3-kserve-575c9774bc-4bmzs\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") " pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:01:58.150610 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.150471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-model-cache\") pod \"auth-test-model-cf82b3-kserve-575c9774bc-4bmzs\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") " pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:01:58.150610 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.150493    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-tmp-dir\") pod \"auth-test-model-cf82b3-kserve-575c9774bc-4bmzs\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") " pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:01:58.150610 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.150551    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bf5f637-1082-42f9-b0ba-dd0f3b274956-tls-certs\") pod \"auth-test-model-cf82b3-kserve-575c9774bc-4bmzs\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") " pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:01:58.150811 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.150617    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-dshm\") pod \"auth-test-model-cf82b3-kserve-575c9774bc-4bmzs\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") " pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:01:58.150945 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.150921    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-home\") pod \"auth-test-model-cf82b3-kserve-575c9774bc-4bmzs\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") " pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:01:58.151011 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.150944    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-model-cache\") pod \"auth-test-model-cf82b3-kserve-575c9774bc-4bmzs\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") " pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:01:58.151011 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.150973    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-tmp-dir\") pod \"auth-test-model-cf82b3-kserve-575c9774bc-4bmzs\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") " pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:01:58.152918 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.152894    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-dshm\") pod \"auth-test-model-cf82b3-kserve-575c9774bc-4bmzs\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") " pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:01:58.153068 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.153054    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bf5f637-1082-42f9-b0ba-dd0f3b274956-tls-certs\") pod \"auth-test-model-cf82b3-kserve-575c9774bc-4bmzs\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") " pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:01:58.159740 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.159717    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7lg6\" (UniqueName: \"kubernetes.io/projected/8bf5f637-1082-42f9-b0ba-dd0f3b274956-kube-api-access-r7lg6\") pod \"auth-test-model-cf82b3-kserve-575c9774bc-4bmzs\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") " pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:01:58.303758 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.303696    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:01:58.441680 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:58.441620    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"]
Jul 22 03:01:58.445806 ip-10-0-140-200 kubenswrapper[2565]: W0722 03:01:58.445767    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bf5f637_1082_42f9_b0ba_dd0f3b274956.slice/crio-9e7d362e85748ba49d12cefbfff3d2265e898b86644fc6eeb952d6fa07b9b89c WatchSource:0}: Error finding container 9e7d362e85748ba49d12cefbfff3d2265e898b86644fc6eeb952d6fa07b9b89c: Status 404 returned error can't find the container with id 9e7d362e85748ba49d12cefbfff3d2265e898b86644fc6eeb952d6fa07b9b89c
Jul 22 03:01:59.031476 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:59.031433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs" event={"ID":"8bf5f637-1082-42f9-b0ba-dd0f3b274956","Type":"ContainerStarted","Data":"d60d36b67075290c5f77b32750578e10de6cd16b89a5ffa6d1969b438641ee57"}
Jul 22 03:01:59.031476 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:59.031480    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs" event={"ID":"8bf5f637-1082-42f9-b0ba-dd0f3b274956","Type":"ContainerStarted","Data":"9e7d362e85748ba49d12cefbfff3d2265e898b86644fc6eeb952d6fa07b9b89c"}
Jul 22 03:01:59.054104 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:01:59.054052    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs" podStartSLOduration=2.054036523 podStartE2EDuration="2.054036523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:01:59.052194143 +0000 UTC m=+2469.900530717" watchObservedRunningTime="2026-07-22 03:01:59.054036523 +0000 UTC m=+2469.902373141"
Jul 22 03:02:08.304692 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:02:08.304654    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:02:08.305307 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:02:08.304705    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:02:08.317037 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:02:08.317010    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:02:09.097087 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:02:09.097050    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:02:23.650070 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:02:23.650036    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:02:42.463946 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:02:42.463840    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:03:50.167400 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:03:50.167363    2565 scope.go:117] "RemoveContainer" containerID="c7c2d30d2086f4bdcb42a700f516efc4356af1efa32b24a966d54ca3068e6a68"
Jul 22 03:03:54.654665 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:03:54.654631    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:03:55.935469 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:03:55.935439    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:03:56.689205 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:03:56.689176    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:03:57.478824 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:03:57.478792    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"]
Jul 22 03:03:57.501382 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:03:57.501339    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:04:09.094625 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.094594    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"]
Jul 22 03:04:09.098043 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.098021    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:09.101629 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.101593    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d8076c\"/\"default-dockercfg-9psq7\""
Jul 22 03:04:09.101752 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.101602    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d8076c\"/\"openshift-service-ca.crt\""
Jul 22 03:04:09.101752 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.101672    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d8076c\"/\"sub-test-model-d8076c-kserve-self-signed-certs\""
Jul 22 03:04:09.103294 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.103274    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d8076c\"/\"kube-root-ca.crt\""
Jul 22 03:04:09.111338 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.111314    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"]
Jul 22 03:04:09.122130 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.122112    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:04:09.183036 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.183002    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-model-cache\") pod \"sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") " pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:09.183210 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.183052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9337becf-0287-4b3f-a667-5c5d263662a1-tls-certs\") pod \"sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") " pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:09.183210 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.183146    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-home\") pod \"sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") " pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:09.183210 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.183169    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-tmp-dir\") pod \"sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") " pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:09.183377 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.183252    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm64r\" (UniqueName: \"kubernetes.io/projected/9337becf-0287-4b3f-a667-5c5d263662a1-kube-api-access-xm64r\") pod \"sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") " pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:09.183377 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.183289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-dshm\") pod \"sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") " pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:09.284166 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.284135    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-model-cache\") pod \"sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") " pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:09.284166 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.284174    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9337becf-0287-4b3f-a667-5c5d263662a1-tls-certs\") pod \"sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") " pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:09.284418 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.284303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-home\") pod \"sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") " pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:09.284418 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.284338    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-tmp-dir\") pod \"sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") " pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:09.284534 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.284417    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm64r\" (UniqueName: \"kubernetes.io/projected/9337becf-0287-4b3f-a667-5c5d263662a1-kube-api-access-xm64r\") pod \"sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") " pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:09.284534 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.284457    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-dshm\") pod \"sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") " pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:09.284640 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.284562    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-model-cache\") pod \"sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") " pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:09.284640 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.284624    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-home\") pod \"sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") " pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:09.284746 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.284693    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-tmp-dir\") pod \"sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") " pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:09.286582 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.286562    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-dshm\") pod \"sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") " pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:09.286729 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.286712    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9337becf-0287-4b3f-a667-5c5d263662a1-tls-certs\") pod \"sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") " pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:09.293094 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.293072    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm64r\" (UniqueName: \"kubernetes.io/projected/9337becf-0287-4b3f-a667-5c5d263662a1-kube-api-access-xm64r\") pod \"sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") " pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:09.408109 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.408076    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:09.545443 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.545392    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"]
Jul 22 03:04:09.547921 ip-10-0-140-200 kubenswrapper[2565]: W0722 03:04:09.547891    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9337becf_0287_4b3f_a667_5c5d263662a1.slice/crio-91e6956112c53adf3d4e21b4d4c55777d12273784a0a6b9ff74f3e4e69ad11b4 WatchSource:0}: Error finding container 91e6956112c53adf3d4e21b4d4c55777d12273784a0a6b9ff74f3e4e69ad11b4: Status 404 returned error can't find the container with id 91e6956112c53adf3d4e21b4d4c55777d12273784a0a6b9ff74f3e4e69ad11b4
Jul 22 03:04:09.691647 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.691555    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb" event={"ID":"9337becf-0287-4b3f-a667-5c5d263662a1","Type":"ContainerStarted","Data":"0c97bae00c7bd9c83a1651822d02c6b9a0e3119174a94cc0c46b2a2597a28341"}
Jul 22 03:04:09.691647 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.691605    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb" event={"ID":"9337becf-0287-4b3f-a667-5c5d263662a1","Type":"ContainerStarted","Data":"91e6956112c53adf3d4e21b4d4c55777d12273784a0a6b9ff74f3e4e69ad11b4"}
Jul 22 03:04:09.711632 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:09.711580    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb" podStartSLOduration=0.71156592 podStartE2EDuration="711.56592ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:04:09.709293021 +0000 UTC m=+2600.557629595" watchObservedRunningTime="2026-07-22 03:04:09.71156592 +0000 UTC m=+2600.559902521"
Jul 22 03:04:12.508605 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:12.508521    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs" podUID="8bf5f637-1082-42f9-b0ba-dd0f3b274956" containerName="main" containerID="cri-o://d60d36b67075290c5f77b32750578e10de6cd16b89a5ffa6d1969b438641ee57" gracePeriod=45
Jul 22 03:04:12.708097 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:12.708065    2565 generic.go:356] "Generic (PLEG): container finished" podID="8bf5f637-1082-42f9-b0ba-dd0f3b274956" containerID="d60d36b67075290c5f77b32750578e10de6cd16b89a5ffa6d1969b438641ee57" exitCode=0
Jul 22 03:04:12.708268 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:12.708106    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs" event={"ID":"8bf5f637-1082-42f9-b0ba-dd0f3b274956","Type":"ContainerDied","Data":"d60d36b67075290c5f77b32750578e10de6cd16b89a5ffa6d1969b438641ee57"}
Jul 22 03:04:12.767103 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:12.767046    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:04:12.918585 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:12.918555    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-tmp-dir\") pod \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") "
Jul 22 03:04:12.918763 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:12.918617    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-dshm\") pod \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") "
Jul 22 03:04:12.918763 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:12.918651    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-home\") pod \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") "
Jul 22 03:04:12.918763 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:12.918677    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bf5f637-1082-42f9-b0ba-dd0f3b274956-tls-certs\") pod \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") "
Jul 22 03:04:12.918763 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:12.918738    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-model-cache\") pod \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") "
Jul 22 03:04:12.919009 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:12.918771    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7lg6\" (UniqueName: \"kubernetes.io/projected/8bf5f637-1082-42f9-b0ba-dd0f3b274956-kube-api-access-r7lg6\") pod \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\" (UID: \"8bf5f637-1082-42f9-b0ba-dd0f3b274956\") "
Jul 22 03:04:12.919009 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:12.918899    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8bf5f637-1082-42f9-b0ba-dd0f3b274956" (UID: "8bf5f637-1082-42f9-b0ba-dd0f3b274956"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:04:12.919119 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:12.919001    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-home" (OuterVolumeSpecName: "home") pod "8bf5f637-1082-42f9-b0ba-dd0f3b274956" (UID: "8bf5f637-1082-42f9-b0ba-dd0f3b274956"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:04:12.919119 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:12.919112    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-tmp-dir\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:04:12.919202 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:12.919130    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-home\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:04:12.919383 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:12.919362    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-model-cache" (OuterVolumeSpecName: "model-cache") pod "8bf5f637-1082-42f9-b0ba-dd0f3b274956" (UID: "8bf5f637-1082-42f9-b0ba-dd0f3b274956"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:04:12.921178 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:12.921152    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf5f637-1082-42f9-b0ba-dd0f3b274956-kube-api-access-r7lg6" (OuterVolumeSpecName: "kube-api-access-r7lg6") pod "8bf5f637-1082-42f9-b0ba-dd0f3b274956" (UID: "8bf5f637-1082-42f9-b0ba-dd0f3b274956"). InnerVolumeSpecName "kube-api-access-r7lg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:04:12.921286 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:12.921259    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf5f637-1082-42f9-b0ba-dd0f3b274956-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8bf5f637-1082-42f9-b0ba-dd0f3b274956" (UID: "8bf5f637-1082-42f9-b0ba-dd0f3b274956"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 03:04:12.921286 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:12.921263    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-dshm" (OuterVolumeSpecName: "dshm") pod "8bf5f637-1082-42f9-b0ba-dd0f3b274956" (UID: "8bf5f637-1082-42f9-b0ba-dd0f3b274956"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:04:13.019957 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:13.019887    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-dshm\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:04:13.019957 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:13.019913    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bf5f637-1082-42f9-b0ba-dd0f3b274956-tls-certs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:04:13.019957 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:13.019923    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bf5f637-1082-42f9-b0ba-dd0f3b274956-model-cache\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:04:13.019957 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:13.019932    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7lg6\" (UniqueName: \"kubernetes.io/projected/8bf5f637-1082-42f9-b0ba-dd0f3b274956-kube-api-access-r7lg6\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:04:13.719224 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:13.719197    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"
Jul 22 03:04:13.719628 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:13.719238    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs" event={"ID":"8bf5f637-1082-42f9-b0ba-dd0f3b274956","Type":"ContainerDied","Data":"9e7d362e85748ba49d12cefbfff3d2265e898b86644fc6eeb952d6fa07b9b89c"}
Jul 22 03:04:13.719628 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:13.719288    2565 scope.go:117] "RemoveContainer" containerID="d60d36b67075290c5f77b32750578e10de6cd16b89a5ffa6d1969b438641ee57"
Jul 22 03:04:13.743192 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:13.743164    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"]
Jul 22 03:04:13.747253 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:13.747233    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-cf82b3/auth-test-model-cf82b3-kserve-575c9774bc-4bmzs"]
Jul 22 03:04:15.726228 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:15.726185    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bf5f637-1082-42f9-b0ba-dd0f3b274956" path="/var/lib/kubelet/pods/8bf5f637-1082-42f9-b0ba-dd0f3b274956/volumes"
Jul 22 03:04:19.408285 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:19.408251    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:19.408761 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:19.408321    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:19.420569 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:19.420545    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:19.758012 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:19.757934    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:04:38.617250 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:04:38.617214    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:05:09.237559 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:09.237513    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:05:23.467479 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:23.467442    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"]
Jul 22 03:05:23.491335 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:23.491306    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:05:24.318616 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:24.318559    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:05:28.860461 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:28.860428    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"]
Jul 22 03:05:28.860958 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:28.860786    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf5f637-1082-42f9-b0ba-dd0f3b274956" containerName="main"
Jul 22 03:05:28.860958 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:28.860796    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf5f637-1082-42f9-b0ba-dd0f3b274956" containerName="main"
Jul 22 03:05:28.860958 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:28.860900    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf5f637-1082-42f9-b0ba-dd0f3b274956" containerName="main"
Jul 22 03:05:28.863957 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:28.863939    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:28.866606 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:28.866579    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d8076c\"/\"rate-test-model-d8076c-kserve-self-signed-certs\""
Jul 22 03:05:28.877286 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:28.877262    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"]
Jul 22 03:05:28.901395 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:28.901374    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:05:28.957051 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:28.957027    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-tmp-dir\") pod \"rate-test-model-d8076c-kserve-5965b6b48c-2th7l\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") " pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:28.957157 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:28.957064    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-tls-certs\") pod \"rate-test-model-d8076c-kserve-5965b6b48c-2th7l\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") " pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:28.957157 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:28.957111    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-dshm\") pod \"rate-test-model-d8076c-kserve-5965b6b48c-2th7l\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") " pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:28.957157 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:28.957132    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-home\") pod \"rate-test-model-d8076c-kserve-5965b6b48c-2th7l\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") " pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:28.957269 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:28.957155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-model-cache\") pod \"rate-test-model-d8076c-kserve-5965b6b48c-2th7l\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") " pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:28.957269 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:28.957212    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k87vg\" (UniqueName: \"kubernetes.io/projected/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-kube-api-access-k87vg\") pod \"rate-test-model-d8076c-kserve-5965b6b48c-2th7l\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") " pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:29.058591 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:29.058557    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-tls-certs\") pod \"rate-test-model-d8076c-kserve-5965b6b48c-2th7l\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") " pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:29.058751 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:29.058618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-dshm\") pod \"rate-test-model-d8076c-kserve-5965b6b48c-2th7l\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") " pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:29.058751 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:29.058641    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-home\") pod \"rate-test-model-d8076c-kserve-5965b6b48c-2th7l\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") " pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:29.058751 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:29.058667    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-model-cache\") pod \"rate-test-model-d8076c-kserve-5965b6b48c-2th7l\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") " pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:29.058751 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:29.058701    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k87vg\" (UniqueName: \"kubernetes.io/projected/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-kube-api-access-k87vg\") pod \"rate-test-model-d8076c-kserve-5965b6b48c-2th7l\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") " pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:29.058751 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:29.058745    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-tmp-dir\") pod \"rate-test-model-d8076c-kserve-5965b6b48c-2th7l\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") " pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:29.059168 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:29.059139    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-home\") pod \"rate-test-model-d8076c-kserve-5965b6b48c-2th7l\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") " pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:29.059275 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:29.059148    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-tmp-dir\") pod \"rate-test-model-d8076c-kserve-5965b6b48c-2th7l\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") " pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:29.059275 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:29.059200    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-model-cache\") pod \"rate-test-model-d8076c-kserve-5965b6b48c-2th7l\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") " pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:29.060919 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:29.060896    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-dshm\") pod \"rate-test-model-d8076c-kserve-5965b6b48c-2th7l\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") " pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:29.061133 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:29.061112    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-tls-certs\") pod \"rate-test-model-d8076c-kserve-5965b6b48c-2th7l\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") " pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:29.067247 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:29.067224    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k87vg\" (UniqueName: \"kubernetes.io/projected/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-kube-api-access-k87vg\") pod \"rate-test-model-d8076c-kserve-5965b6b48c-2th7l\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") " pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:29.174971 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:29.174909    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:29.307841 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:29.307817    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"]
Jul 22 03:05:29.309776 ip-10-0-140-200 kubenswrapper[2565]: W0722 03:05:29.309746    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e66f2aa_ba7f_4a24_b744_cf815dcca7ae.slice/crio-e54d09f02e7e01afcf3fc7e96d1ba5c4652491181c9991382530313feafcca10 WatchSource:0}: Error finding container e54d09f02e7e01afcf3fc7e96d1ba5c4652491181c9991382530313feafcca10: Status 404 returned error can't find the container with id e54d09f02e7e01afcf3fc7e96d1ba5c4652491181c9991382530313feafcca10
Jul 22 03:05:30.073149 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:30.073116    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l" event={"ID":"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae","Type":"ContainerStarted","Data":"f8dd40be17485ae4543f949d794e3125611e72ded486d078dab0bacecdac5ec8"}
Jul 22 03:05:30.073508 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:30.073158    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l" event={"ID":"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae","Type":"ContainerStarted","Data":"e54d09f02e7e01afcf3fc7e96d1ba5c4652491181c9991382530313feafcca10"}
Jul 22 03:05:30.091343 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:30.091291    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l" podStartSLOduration=2.091277301 podStartE2EDuration="2.091277301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:05:30.090437353 +0000 UTC m=+2680.938773928" watchObservedRunningTime="2026-07-22 03:05:30.091277301 +0000 UTC m=+2680.939613874"
Jul 22 03:05:38.174533 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.174504    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:05:38.292150 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.292112    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:05:38.487696 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.487607    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb" podUID="9337becf-0287-4b3f-a667-5c5d263662a1" containerName="main" containerID="cri-o://0c97bae00c7bd9c83a1651822d02c6b9a0e3119174a94cc0c46b2a2597a28341" gracePeriod=45
Jul 22 03:05:38.740968 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.740906    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:05:38.848818 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.848783    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm64r\" (UniqueName: \"kubernetes.io/projected/9337becf-0287-4b3f-a667-5c5d263662a1-kube-api-access-xm64r\") pod \"9337becf-0287-4b3f-a667-5c5d263662a1\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") "
Jul 22 03:05:38.849009 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.848821    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-dshm\") pod \"9337becf-0287-4b3f-a667-5c5d263662a1\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") "
Jul 22 03:05:38.849009 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.848873    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-home\") pod \"9337becf-0287-4b3f-a667-5c5d263662a1\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") "
Jul 22 03:05:38.849009 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.848962    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-model-cache\") pod \"9337becf-0287-4b3f-a667-5c5d263662a1\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") "
Jul 22 03:05:38.849009 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.848985    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9337becf-0287-4b3f-a667-5c5d263662a1-tls-certs\") pod \"9337becf-0287-4b3f-a667-5c5d263662a1\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") "
Jul 22 03:05:38.849228 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.849040    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-tmp-dir\") pod \"9337becf-0287-4b3f-a667-5c5d263662a1\" (UID: \"9337becf-0287-4b3f-a667-5c5d263662a1\") "
Jul 22 03:05:38.849282 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.849227    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-home" (OuterVolumeSpecName: "home") pod "9337becf-0287-4b3f-a667-5c5d263662a1" (UID: "9337becf-0287-4b3f-a667-5c5d263662a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:05:38.849282 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.849236    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "9337becf-0287-4b3f-a667-5c5d263662a1" (UID: "9337becf-0287-4b3f-a667-5c5d263662a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:05:38.849385 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.849350    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-home\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:05:38.849385 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.849367    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-model-cache\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:05:38.849385 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.849343    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9337becf-0287-4b3f-a667-5c5d263662a1" (UID: "9337becf-0287-4b3f-a667-5c5d263662a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:05:38.851091 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.851061    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9337becf-0287-4b3f-a667-5c5d263662a1-kube-api-access-xm64r" (OuterVolumeSpecName: "kube-api-access-xm64r") pod "9337becf-0287-4b3f-a667-5c5d263662a1" (UID: "9337becf-0287-4b3f-a667-5c5d263662a1"). InnerVolumeSpecName "kube-api-access-xm64r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:05:38.851191 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.851093    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9337becf-0287-4b3f-a667-5c5d263662a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9337becf-0287-4b3f-a667-5c5d263662a1" (UID: "9337becf-0287-4b3f-a667-5c5d263662a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 03:05:38.851191 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.851132    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-dshm" (OuterVolumeSpecName: "dshm") pod "9337becf-0287-4b3f-a667-5c5d263662a1" (UID: "9337becf-0287-4b3f-a667-5c5d263662a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:05:38.950100 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.950068    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm64r\" (UniqueName: \"kubernetes.io/projected/9337becf-0287-4b3f-a667-5c5d263662a1-kube-api-access-xm64r\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:05:38.950100 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.950093    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-dshm\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:05:38.950281 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.950107    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9337becf-0287-4b3f-a667-5c5d263662a1-tls-certs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:05:38.950281 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:38.950119    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9337becf-0287-4b3f-a667-5c5d263662a1-tmp-dir\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:05:39.118229 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:39.118181    2565 generic.go:356] "Generic (PLEG): container finished" podID="9337becf-0287-4b3f-a667-5c5d263662a1" containerID="0c97bae00c7bd9c83a1651822d02c6b9a0e3119174a94cc0c46b2a2597a28341" exitCode=0
Jul 22 03:05:39.118377 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:39.118255    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"
Jul 22 03:05:39.118377 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:39.118270    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb" event={"ID":"9337becf-0287-4b3f-a667-5c5d263662a1","Type":"ContainerDied","Data":"0c97bae00c7bd9c83a1651822d02c6b9a0e3119174a94cc0c46b2a2597a28341"}
Jul 22 03:05:39.118377 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:39.118325    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb" event={"ID":"9337becf-0287-4b3f-a667-5c5d263662a1","Type":"ContainerDied","Data":"91e6956112c53adf3d4e21b4d4c55777d12273784a0a6b9ff74f3e4e69ad11b4"}
Jul 22 03:05:39.118377 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:39.118348    2565 scope.go:117] "RemoveContainer" containerID="0c97bae00c7bd9c83a1651822d02c6b9a0e3119174a94cc0c46b2a2597a28341"
Jul 22 03:05:39.127717 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:39.127702    2565 scope.go:117] "RemoveContainer" containerID="0c97bae00c7bd9c83a1651822d02c6b9a0e3119174a94cc0c46b2a2597a28341"
Jul 22 03:05:39.127996 ip-10-0-140-200 kubenswrapper[2565]: E0722 03:05:39.127977    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c97bae00c7bd9c83a1651822d02c6b9a0e3119174a94cc0c46b2a2597a28341\": container with ID starting with 0c97bae00c7bd9c83a1651822d02c6b9a0e3119174a94cc0c46b2a2597a28341 not found: ID does not exist" containerID="0c97bae00c7bd9c83a1651822d02c6b9a0e3119174a94cc0c46b2a2597a28341"
Jul 22 03:05:39.128077 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:39.128001    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c97bae00c7bd9c83a1651822d02c6b9a0e3119174a94cc0c46b2a2597a28341"} err="failed to get container status \"0c97bae00c7bd9c83a1651822d02c6b9a0e3119174a94cc0c46b2a2597a28341\": rpc error: code = NotFound desc = could not find container \"0c97bae00c7bd9c83a1651822d02c6b9a0e3119174a94cc0c46b2a2597a28341\": container with ID starting with 0c97bae00c7bd9c83a1651822d02c6b9a0e3119174a94cc0c46b2a2597a28341 not found: ID does not exist"
Jul 22 03:05:39.141704 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:39.141684    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"]
Jul 22 03:05:39.147326 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:39.147306    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d8076c/sub-test-model-d8076c-kserve-7df4d5c55f-8w4sb"]
Jul 22 03:05:39.175683 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:39.175662    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:39.175989 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:39.175881    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:39.188270 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:39.188252    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:39.726006 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:39.725974    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9337becf-0287-4b3f-a667-5c5d263662a1" path="/var/lib/kubelet/pods/9337becf-0287-4b3f-a667-5c5d263662a1/volumes"
Jul 22 03:05:40.136305 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:40.136274    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:05:43.585580 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:43.585498    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:05:50.202708 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:50.202670    2565 scope.go:117] "RemoveContainer" containerID="ce58f459e4283576b323a1078998226aa38160545c1eafe1735cadd5c0b98324"
Jul 22 03:05:58.249007 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:58.248976    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:05:58.352760 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:05:58.352708    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:06:05.928308 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:06:05.928269    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:06:06.060379 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:06:06.060347    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:06:18.716427 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:06:18.716387    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:06:50.226116 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:06:50.226084    2565 scope.go:117] "RemoveContainer" containerID="42becd366569073d1838252addee0e931e9782394c5cce30b2905591290da847"
Jul 22 03:06:58.697174 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:06:58.697140    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"]
Jul 22 03:06:58.715576 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:06:58.715540    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:06:59.579999 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:06:59.579959    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:07:00.528576 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:00.528545    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7730-140712170496560"]
Jul 22 03:07:00.528957 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:00.528941    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9337becf-0287-4b3f-a667-5c5d263662a1" containerName="main"
Jul 22 03:07:00.528957 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:00.528958    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9337becf-0287-4b3f-a667-5c5d263662a1" containerName="main"
Jul 22 03:07:00.529036 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:00.529014    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9337becf-0287-4b3f-a667-5c5d263662a1" containerName="main"
Jul 22 03:07:00.531748 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:00.531731    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7730-140712170496560"
Jul 22 03:07:00.534214 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:00.534182    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-88sqm\""
Jul 22 03:07:00.539627 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:00.539605    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7730-140712170496560"]
Jul 22 03:07:00.594171 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:00.594141    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn8xf\" (UniqueName: \"kubernetes.io/projected/298fab39-843d-4f6c-adbf-dd854b4a5169-kube-api-access-kn8xf\") pod \"test-curl-7730-140712170496560\" (UID: \"298fab39-843d-4f6c-adbf-dd854b4a5169\") " pod="opendatahub/test-curl-7730-140712170496560"
Jul 22 03:07:00.695656 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:00.695628    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn8xf\" (UniqueName: \"kubernetes.io/projected/298fab39-843d-4f6c-adbf-dd854b4a5169-kube-api-access-kn8xf\") pod \"test-curl-7730-140712170496560\" (UID: \"298fab39-843d-4f6c-adbf-dd854b4a5169\") " pod="opendatahub/test-curl-7730-140712170496560"
Jul 22 03:07:00.703955 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:00.703933    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn8xf\" (UniqueName: \"kubernetes.io/projected/298fab39-843d-4f6c-adbf-dd854b4a5169-kube-api-access-kn8xf\") pod \"test-curl-7730-140712170496560\" (UID: \"298fab39-843d-4f6c-adbf-dd854b4a5169\") " pod="opendatahub/test-curl-7730-140712170496560"
Jul 22 03:07:00.841691 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:00.841601    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7730-140712170496560"
Jul 22 03:07:00.976257 ip-10-0-140-200 kubenswrapper[2565]: W0722 03:07:00.976230    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod298fab39_843d_4f6c_adbf_dd854b4a5169.slice/crio-20388482f9ae37aa81f37cced1665fe11f6f72b1d888a2fb7083e1da39cb5f28 WatchSource:0}: Error finding container 20388482f9ae37aa81f37cced1665fe11f6f72b1d888a2fb7083e1da39cb5f28: Status 404 returned error can't find the container with id 20388482f9ae37aa81f37cced1665fe11f6f72b1d888a2fb7083e1da39cb5f28
Jul 22 03:07:00.977340 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:00.977314    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7730-140712170496560"]
Jul 22 03:07:00.977577 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:00.977563    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 03:07:01.505898 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:01.505842    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7730-140712170496560" event={"ID":"298fab39-843d-4f6c-adbf-dd854b4a5169","Type":"ContainerStarted","Data":"20388482f9ae37aa81f37cced1665fe11f6f72b1d888a2fb7083e1da39cb5f28"}
Jul 22 03:07:02.512831 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:02.512799    2565 generic.go:356] "Generic (PLEG): container finished" podID="298fab39-843d-4f6c-adbf-dd854b4a5169" containerID="41a1e0364eb4efa73d558683f8d8ca498ba6d49321fb1d1b8157e45f58cb6601" exitCode=0
Jul 22 03:07:02.513216 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:02.512869    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7730-140712170496560" event={"ID":"298fab39-843d-4f6c-adbf-dd854b4a5169","Type":"ContainerDied","Data":"41a1e0364eb4efa73d558683f8d8ca498ba6d49321fb1d1b8157e45f58cb6601"}
Jul 22 03:07:03.659147 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:03.659124    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7730-140712170496560"
Jul 22 03:07:03.680932 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:03.680901    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7730-140712170496560_298fab39-843d-4f6c-adbf-dd854b4a5169/test-curl-7730-140712170496560/0.log"
Jul 22 03:07:03.701948 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:03.701924    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7730-140712170496560"]
Jul 22 03:07:03.705810 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:03.705787    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7730-140712170496560"]
Jul 22 03:07:03.722379 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:03.722363    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn8xf\" (UniqueName: \"kubernetes.io/projected/298fab39-843d-4f6c-adbf-dd854b4a5169-kube-api-access-kn8xf\") pod \"298fab39-843d-4f6c-adbf-dd854b4a5169\" (UID: \"298fab39-843d-4f6c-adbf-dd854b4a5169\") "
Jul 22 03:07:03.724572 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:03.724543    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298fab39-843d-4f6c-adbf-dd854b4a5169-kube-api-access-kn8xf" (OuterVolumeSpecName: "kube-api-access-kn8xf") pod "298fab39-843d-4f6c-adbf-dd854b4a5169" (UID: "298fab39-843d-4f6c-adbf-dd854b4a5169"). InnerVolumeSpecName "kube-api-access-kn8xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:07:03.726692 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:03.726671    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="298fab39-843d-4f6c-adbf-dd854b4a5169" path="/var/lib/kubelet/pods/298fab39-843d-4f6c-adbf-dd854b4a5169/volumes"
Jul 22 03:07:03.824027 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:03.823951    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn8xf\" (UniqueName: \"kubernetes.io/projected/298fab39-843d-4f6c-adbf-dd854b4a5169-kube-api-access-kn8xf\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:03.829191 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:03.829168    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7730-140712170498864"]
Jul 22 03:07:03.829646 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:03.829629    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="298fab39-843d-4f6c-adbf-dd854b4a5169" containerName="test-curl-7730-140712170496560"
Jul 22 03:07:03.829741 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:03.829648    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="298fab39-843d-4f6c-adbf-dd854b4a5169" containerName="test-curl-7730-140712170496560"
Jul 22 03:07:03.829794 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:03.829739    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="298fab39-843d-4f6c-adbf-dd854b4a5169" containerName="test-curl-7730-140712170496560"
Jul 22 03:07:03.832779 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:03.832762    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7730-140712170498864"
Jul 22 03:07:03.839075 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:03.839055    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7730-140712170498864"]
Jul 22 03:07:03.925069 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:03.925039    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhgwm\" (UniqueName: \"kubernetes.io/projected/441668f0-3dfe-47a3-84f7-85127780ccc4-kube-api-access-xhgwm\") pod \"test-curl-7730-140712170498864\" (UID: \"441668f0-3dfe-47a3-84f7-85127780ccc4\") " pod="opendatahub/test-curl-7730-140712170498864"
Jul 22 03:07:04.026306 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:04.026274    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhgwm\" (UniqueName: \"kubernetes.io/projected/441668f0-3dfe-47a3-84f7-85127780ccc4-kube-api-access-xhgwm\") pod \"test-curl-7730-140712170498864\" (UID: \"441668f0-3dfe-47a3-84f7-85127780ccc4\") " pod="opendatahub/test-curl-7730-140712170498864"
Jul 22 03:07:04.035253 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:04.035224    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhgwm\" (UniqueName: \"kubernetes.io/projected/441668f0-3dfe-47a3-84f7-85127780ccc4-kube-api-access-xhgwm\") pod \"test-curl-7730-140712170498864\" (UID: \"441668f0-3dfe-47a3-84f7-85127780ccc4\") " pod="opendatahub/test-curl-7730-140712170498864"
Jul 22 03:07:04.144210 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:04.144187    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7730-140712170498864"
Jul 22 03:07:04.275480 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:04.275457    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7730-140712170498864"]
Jul 22 03:07:04.277177 ip-10-0-140-200 kubenswrapper[2565]: W0722 03:07:04.277144    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod441668f0_3dfe_47a3_84f7_85127780ccc4.slice/crio-bc345ff6f9352ffbb22de67b9317f2f2db7659130bdec63556cc5bb830d41362 WatchSource:0}: Error finding container bc345ff6f9352ffbb22de67b9317f2f2db7659130bdec63556cc5bb830d41362: Status 404 returned error can't find the container with id bc345ff6f9352ffbb22de67b9317f2f2db7659130bdec63556cc5bb830d41362
Jul 22 03:07:04.526679 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:04.526600    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7730-140712170496560"
Jul 22 03:07:04.526679 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:04.526657    2565 scope.go:117] "RemoveContainer" containerID="41a1e0364eb4efa73d558683f8d8ca498ba6d49321fb1d1b8157e45f58cb6601"
Jul 22 03:07:04.527993 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:04.527920    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7730-140712170498864" event={"ID":"441668f0-3dfe-47a3-84f7-85127780ccc4","Type":"ContainerStarted","Data":"bc345ff6f9352ffbb22de67b9317f2f2db7659130bdec63556cc5bb830d41362"}
Jul 22 03:07:05.535229 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:05.535189    2565 generic.go:356] "Generic (PLEG): container finished" podID="441668f0-3dfe-47a3-84f7-85127780ccc4" containerID="12502de7e350b7b52e3aa52340870c332c0d5e88f249b1bc97689d5d49fee134" exitCode=0
Jul 22 03:07:05.535644 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:05.535269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7730-140712170498864" event={"ID":"441668f0-3dfe-47a3-84f7-85127780ccc4","Type":"ContainerDied","Data":"12502de7e350b7b52e3aa52340870c332c0d5e88f249b1bc97689d5d49fee134"}
Jul 22 03:07:06.683261 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:06.683238    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7730-140712170498864"
Jul 22 03:07:06.704434 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:06.704406    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7730-140712170498864_441668f0-3dfe-47a3-84f7-85127780ccc4/test-curl-7730-140712170498864/0.log"
Jul 22 03:07:06.726568 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:06.726542    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7730-140712170498864"]
Jul 22 03:07:06.729433 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:06.729410    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7730-140712170498864"]
Jul 22 03:07:06.749651 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:06.749618    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhgwm\" (UniqueName: \"kubernetes.io/projected/441668f0-3dfe-47a3-84f7-85127780ccc4-kube-api-access-xhgwm\") pod \"441668f0-3dfe-47a3-84f7-85127780ccc4\" (UID: \"441668f0-3dfe-47a3-84f7-85127780ccc4\") "
Jul 22 03:07:06.752074 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:06.752042    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441668f0-3dfe-47a3-84f7-85127780ccc4-kube-api-access-xhgwm" (OuterVolumeSpecName: "kube-api-access-xhgwm") pod "441668f0-3dfe-47a3-84f7-85127780ccc4" (UID: "441668f0-3dfe-47a3-84f7-85127780ccc4"). InnerVolumeSpecName "kube-api-access-xhgwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:07:06.851449 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:06.851379    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhgwm\" (UniqueName: \"kubernetes.io/projected/441668f0-3dfe-47a3-84f7-85127780ccc4-kube-api-access-xhgwm\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:06.892234 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:06.892208    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7730-140712170337312"]
Jul 22 03:07:06.892600 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:06.892588    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="441668f0-3dfe-47a3-84f7-85127780ccc4" containerName="test-curl-7730-140712170498864"
Jul 22 03:07:06.892644 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:06.892602    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="441668f0-3dfe-47a3-84f7-85127780ccc4" containerName="test-curl-7730-140712170498864"
Jul 22 03:07:06.892679 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:06.892657    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="441668f0-3dfe-47a3-84f7-85127780ccc4" containerName="test-curl-7730-140712170498864"
Jul 22 03:07:06.895539 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:06.895523    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7730-140712170337312"
Jul 22 03:07:06.903401 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:06.903382    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7730-140712170337312"]
Jul 22 03:07:06.951928 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:06.951899    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c8fk\" (UniqueName: \"kubernetes.io/projected/3c9ba3b8-3e5f-41e1-a18c-976668dc30a0-kube-api-access-7c8fk\") pod \"test-curl-7730-140712170337312\" (UID: \"3c9ba3b8-3e5f-41e1-a18c-976668dc30a0\") " pod="opendatahub/test-curl-7730-140712170337312"
Jul 22 03:07:07.053243 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:07.053216    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c8fk\" (UniqueName: \"kubernetes.io/projected/3c9ba3b8-3e5f-41e1-a18c-976668dc30a0-kube-api-access-7c8fk\") pod \"test-curl-7730-140712170337312\" (UID: \"3c9ba3b8-3e5f-41e1-a18c-976668dc30a0\") " pod="opendatahub/test-curl-7730-140712170337312"
Jul 22 03:07:07.061101 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:07.061080    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c8fk\" (UniqueName: \"kubernetes.io/projected/3c9ba3b8-3e5f-41e1-a18c-976668dc30a0-kube-api-access-7c8fk\") pod \"test-curl-7730-140712170337312\" (UID: \"3c9ba3b8-3e5f-41e1-a18c-976668dc30a0\") " pod="opendatahub/test-curl-7730-140712170337312"
Jul 22 03:07:07.205965 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:07.205925    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7730-140712170337312"
Jul 22 03:07:07.333892 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:07.333869    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7730-140712170337312"]
Jul 22 03:07:07.335777 ip-10-0-140-200 kubenswrapper[2565]: W0722 03:07:07.335739    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c9ba3b8_3e5f_41e1_a18c_976668dc30a0.slice/crio-e42a7d5ab886f89fcb719064f999f45ed7eac664f4d5c7c7ae1c781f11ae30dc WatchSource:0}: Error finding container e42a7d5ab886f89fcb719064f999f45ed7eac664f4d5c7c7ae1c781f11ae30dc: Status 404 returned error can't find the container with id e42a7d5ab886f89fcb719064f999f45ed7eac664f4d5c7c7ae1c781f11ae30dc
Jul 22 03:07:07.546963 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:07.546888    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7730-140712170337312" event={"ID":"3c9ba3b8-3e5f-41e1-a18c-976668dc30a0","Type":"ContainerStarted","Data":"e42a7d5ab886f89fcb719064f999f45ed7eac664f4d5c7c7ae1c781f11ae30dc"}
Jul 22 03:07:07.548321 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:07.548300    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc345ff6f9352ffbb22de67b9317f2f2db7659130bdec63556cc5bb830d41362"
Jul 22 03:07:07.548321 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:07.548314    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7730-140712170498864"
Jul 22 03:07:07.558634 ip-10-0-140-200 kubenswrapper[2565]: E0722 03:07:07.558610    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7730-140712170498864\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="441668f0-3dfe-47a3-84f7-85127780ccc4" pod="opendatahub/test-curl-7730-140712170498864"
Jul 22 03:07:07.727048 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:07.727018    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="441668f0-3dfe-47a3-84f7-85127780ccc4" path="/var/lib/kubelet/pods/441668f0-3dfe-47a3-84f7-85127780ccc4/volumes"
Jul 22 03:07:08.555030 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:08.554990    2565 generic.go:356] "Generic (PLEG): container finished" podID="3c9ba3b8-3e5f-41e1-a18c-976668dc30a0" containerID="5be8aa3bb72f80df9833a9acaf87492fa1ec97668508b011e54725b04ed492a9" exitCode=0
Jul 22 03:07:08.555211 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:08.555047    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7730-140712170337312" event={"ID":"3c9ba3b8-3e5f-41e1-a18c-976668dc30a0","Type":"ContainerDied","Data":"5be8aa3bb72f80df9833a9acaf87492fa1ec97668508b011e54725b04ed492a9"}
Jul 22 03:07:09.689235 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:09.689215    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7730-140712170337312"
Jul 22 03:07:09.707761 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:09.707740    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7730-140712170337312_3c9ba3b8-3e5f-41e1-a18c-976668dc30a0/test-curl-7730-140712170337312/0.log"
Jul 22 03:07:09.731146 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:09.731119    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7730-140712170337312"]
Jul 22 03:07:09.735064 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:09.735039    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7730-140712170337312"]
Jul 22 03:07:09.777153 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:09.777127    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7c8fk\" (UniqueName: \"kubernetes.io/projected/3c9ba3b8-3e5f-41e1-a18c-976668dc30a0-kube-api-access-7c8fk\") pod \"3c9ba3b8-3e5f-41e1-a18c-976668dc30a0\" (UID: \"3c9ba3b8-3e5f-41e1-a18c-976668dc30a0\") "
Jul 22 03:07:09.779122 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:09.779099    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9ba3b8-3e5f-41e1-a18c-976668dc30a0-kube-api-access-7c8fk" (OuterVolumeSpecName: "kube-api-access-7c8fk") pod "3c9ba3b8-3e5f-41e1-a18c-976668dc30a0" (UID: "3c9ba3b8-3e5f-41e1-a18c-976668dc30a0"). InnerVolumeSpecName "kube-api-access-7c8fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:07:09.855788 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:09.855764    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7730-140712170337312"]
Jul 22 03:07:09.856145 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:09.856132    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9ba3b8-3e5f-41e1-a18c-976668dc30a0" containerName="test-curl-7730-140712170337312"
Jul 22 03:07:09.856253 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:09.856147    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9ba3b8-3e5f-41e1-a18c-976668dc30a0" containerName="test-curl-7730-140712170337312"
Jul 22 03:07:09.856253 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:09.856223    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c9ba3b8-3e5f-41e1-a18c-976668dc30a0" containerName="test-curl-7730-140712170337312"
Jul 22 03:07:09.859077 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:09.859063    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7730-140712170337312"
Jul 22 03:07:09.865440 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:09.865414    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7730-140712170337312"]
Jul 22 03:07:09.877817 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:09.877792    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c8fk\" (UniqueName: \"kubernetes.io/projected/3c9ba3b8-3e5f-41e1-a18c-976668dc30a0-kube-api-access-7c8fk\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:09.979144 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:09.979114    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvkxt\" (UniqueName: \"kubernetes.io/projected/b4b2e7a9-b6c4-4196-9ab4-5171314ca102-kube-api-access-dvkxt\") pod \"test-curl-7730-140712170337312\" (UID: \"b4b2e7a9-b6c4-4196-9ab4-5171314ca102\") " pod="opendatahub/test-curl-7730-140712170337312"
Jul 22 03:07:10.080331 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:10.080298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvkxt\" (UniqueName: \"kubernetes.io/projected/b4b2e7a9-b6c4-4196-9ab4-5171314ca102-kube-api-access-dvkxt\") pod \"test-curl-7730-140712170337312\" (UID: \"b4b2e7a9-b6c4-4196-9ab4-5171314ca102\") " pod="opendatahub/test-curl-7730-140712170337312"
Jul 22 03:07:10.088932 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:10.088914    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvkxt\" (UniqueName: \"kubernetes.io/projected/b4b2e7a9-b6c4-4196-9ab4-5171314ca102-kube-api-access-dvkxt\") pod \"test-curl-7730-140712170337312\" (UID: \"b4b2e7a9-b6c4-4196-9ab4-5171314ca102\") " pod="opendatahub/test-curl-7730-140712170337312"
Jul 22 03:07:10.170146 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:10.170084    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7730-140712170337312"
Jul 22 03:07:10.299261 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:10.299236    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7730-140712170337312"]
Jul 22 03:07:10.300925 ip-10-0-140-200 kubenswrapper[2565]: W0722 03:07:10.300900    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4b2e7a9_b6c4_4196_9ab4_5171314ca102.slice/crio-56e05accee7103b7ab38d66adb2c9386683bc61a3cd745f1afd9ecfa4a1f30ba WatchSource:0}: Error finding container 56e05accee7103b7ab38d66adb2c9386683bc61a3cd745f1afd9ecfa4a1f30ba: Status 404 returned error can't find the container with id 56e05accee7103b7ab38d66adb2c9386683bc61a3cd745f1afd9ecfa4a1f30ba
Jul 22 03:07:10.565887 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:10.565787    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7730-140712170337312"
Jul 22 03:07:10.565887 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:10.565834    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e42a7d5ab886f89fcb719064f999f45ed7eac664f4d5c7c7ae1c781f11ae30dc"
Jul 22 03:07:10.567130 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:10.567100    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7730-140712170337312" event={"ID":"b4b2e7a9-b6c4-4196-9ab4-5171314ca102","Type":"ContainerStarted","Data":"56e05accee7103b7ab38d66adb2c9386683bc61a3cd745f1afd9ecfa4a1f30ba"}
Jul 22 03:07:10.578064 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:10.578035    2565 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7730-140712170337312" oldPodUID="3c9ba3b8-3e5f-41e1-a18c-976668dc30a0" podUID="b4b2e7a9-b6c4-4196-9ab4-5171314ca102"
Jul 22 03:07:11.573285 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:11.573205    2565 generic.go:356] "Generic (PLEG): container finished" podID="b4b2e7a9-b6c4-4196-9ab4-5171314ca102" containerID="413c137af0a81ec2de931ea3ddbbdbd9267faae8810ce7a1bcac7322c4bef367" exitCode=0
Jul 22 03:07:11.573626 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:11.573288    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7730-140712170337312" event={"ID":"b4b2e7a9-b6c4-4196-9ab4-5171314ca102","Type":"ContainerDied","Data":"413c137af0a81ec2de931ea3ddbbdbd9267faae8810ce7a1bcac7322c4bef367"}
Jul 22 03:07:11.726362 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:11.726320    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c9ba3b8-3e5f-41e1-a18c-976668dc30a0" path="/var/lib/kubelet/pods/3c9ba3b8-3e5f-41e1-a18c-976668dc30a0/volumes"
Jul 22 03:07:12.713303 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:12.713279    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7730-140712170337312"
Jul 22 03:07:12.732538 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:12.732512    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7730-140712170337312_b4b2e7a9-b6c4-4196-9ab4-5171314ca102/test-curl-7730-140712170337312/0.log"
Jul 22 03:07:12.755835 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:12.755810    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7730-140712170337312"]
Jul 22 03:07:12.759582 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:12.759562    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7730-140712170337312"]
Jul 22 03:07:12.806843 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:12.806812    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvkxt\" (UniqueName: \"kubernetes.io/projected/b4b2e7a9-b6c4-4196-9ab4-5171314ca102-kube-api-access-dvkxt\") pod \"b4b2e7a9-b6c4-4196-9ab4-5171314ca102\" (UID: \"b4b2e7a9-b6c4-4196-9ab4-5171314ca102\") "
Jul 22 03:07:12.808829 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:12.808808    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b2e7a9-b6c4-4196-9ab4-5171314ca102-kube-api-access-dvkxt" (OuterVolumeSpecName: "kube-api-access-dvkxt") pod "b4b2e7a9-b6c4-4196-9ab4-5171314ca102" (UID: "b4b2e7a9-b6c4-4196-9ab4-5171314ca102"). InnerVolumeSpecName "kube-api-access-dvkxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:07:12.883702 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:12.883668    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7730-140712170338320"]
Jul 22 03:07:12.884117 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:12.884102    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4b2e7a9-b6c4-4196-9ab4-5171314ca102" containerName="test-curl-7730-140712170337312"
Jul 22 03:07:12.884162 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:12.884119    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4b2e7a9-b6c4-4196-9ab4-5171314ca102" containerName="test-curl-7730-140712170337312"
Jul 22 03:07:12.884201 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:12.884187    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4b2e7a9-b6c4-4196-9ab4-5171314ca102" containerName="test-curl-7730-140712170337312"
Jul 22 03:07:12.887156 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:12.887140    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7730-140712170338320"
Jul 22 03:07:12.893679 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:12.893653    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7730-140712170338320"]
Jul 22 03:07:12.907930 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:12.907904    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvkxt\" (UniqueName: \"kubernetes.io/projected/b4b2e7a9-b6c4-4196-9ab4-5171314ca102-kube-api-access-dvkxt\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:13.009126 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:13.009099    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snpmw\" (UniqueName: \"kubernetes.io/projected/73293410-2066-48cb-9247-c849a0b1f70e-kube-api-access-snpmw\") pod \"test-curl-7730-140712170338320\" (UID: \"73293410-2066-48cb-9247-c849a0b1f70e\") " pod="opendatahub/test-curl-7730-140712170338320"
Jul 22 03:07:13.109899 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:13.109866    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snpmw\" (UniqueName: \"kubernetes.io/projected/73293410-2066-48cb-9247-c849a0b1f70e-kube-api-access-snpmw\") pod \"test-curl-7730-140712170338320\" (UID: \"73293410-2066-48cb-9247-c849a0b1f70e\") " pod="opendatahub/test-curl-7730-140712170338320"
Jul 22 03:07:13.118725 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:13.118703    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snpmw\" (UniqueName: \"kubernetes.io/projected/73293410-2066-48cb-9247-c849a0b1f70e-kube-api-access-snpmw\") pod \"test-curl-7730-140712170338320\" (UID: \"73293410-2066-48cb-9247-c849a0b1f70e\") " pod="opendatahub/test-curl-7730-140712170338320"
Jul 22 03:07:13.197191 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:13.197133    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7730-140712170338320"
Jul 22 03:07:13.332977 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:13.332949    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7730-140712170338320"]
Jul 22 03:07:13.334435 ip-10-0-140-200 kubenswrapper[2565]: W0722 03:07:13.334410    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73293410_2066_48cb_9247_c849a0b1f70e.slice/crio-27ace1177869aeed056e1506f5f9c5ada4a7ed79e39b570b244d55d27522af95 WatchSource:0}: Error finding container 27ace1177869aeed056e1506f5f9c5ada4a7ed79e39b570b244d55d27522af95: Status 404 returned error can't find the container with id 27ace1177869aeed056e1506f5f9c5ada4a7ed79e39b570b244d55d27522af95
Jul 22 03:07:13.585774 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:13.585697    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7730-140712170337312"
Jul 22 03:07:13.585774 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:13.585717    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56e05accee7103b7ab38d66adb2c9386683bc61a3cd745f1afd9ecfa4a1f30ba"
Jul 22 03:07:13.586905 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:13.586878    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7730-140712170338320" event={"ID":"73293410-2066-48cb-9247-c849a0b1f70e","Type":"ContainerStarted","Data":"27ace1177869aeed056e1506f5f9c5ada4a7ed79e39b570b244d55d27522af95"}
Jul 22 03:07:13.596005 ip-10-0-140-200 kubenswrapper[2565]: E0722 03:07:13.595983    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7730-140712170337312\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="b4b2e7a9-b6c4-4196-9ab4-5171314ca102" pod="opendatahub/test-curl-7730-140712170337312"
Jul 22 03:07:13.720579 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:13.720546    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l" podUID="8e66f2aa-ba7f-4a24-b744-cf815dcca7ae" containerName="main" containerID="cri-o://f8dd40be17485ae4543f949d794e3125611e72ded486d078dab0bacecdac5ec8" gracePeriod=45
Jul 22 03:07:13.726298 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:13.726275    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4b2e7a9-b6c4-4196-9ab4-5171314ca102" path="/var/lib/kubelet/pods/b4b2e7a9-b6c4-4196-9ab4-5171314ca102/volumes"
Jul 22 03:07:13.990117 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:13.990096    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:07:14.117763 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.117734    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-tmp-dir\") pod \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") "
Jul 22 03:07:14.117960 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.117780    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-model-cache\") pod \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") "
Jul 22 03:07:14.117960 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.117802    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-dshm\") pod \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") "
Jul 22 03:07:14.117960 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.117889    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-tls-certs\") pod \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") "
Jul 22 03:07:14.117960 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.117921    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-home\") pod \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") "
Jul 22 03:07:14.118190 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.117966    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k87vg\" (UniqueName: \"kubernetes.io/projected/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-kube-api-access-k87vg\") pod \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\" (UID: \"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae\") "
Jul 22 03:07:14.118190 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.118065    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e66f2aa-ba7f-4a24-b744-cf815dcca7ae" (UID: "8e66f2aa-ba7f-4a24-b744-cf815dcca7ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:07:14.118190 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.118075    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e66f2aa-ba7f-4a24-b744-cf815dcca7ae" (UID: "8e66f2aa-ba7f-4a24-b744-cf815dcca7ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:07:14.118341 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.118229    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-tmp-dir\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:14.118341 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.118243    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-model-cache\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:14.118341 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.118248    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-home" (OuterVolumeSpecName: "home") pod "8e66f2aa-ba7f-4a24-b744-cf815dcca7ae" (UID: "8e66f2aa-ba7f-4a24-b744-cf815dcca7ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:07:14.120109 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.120080    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-dshm" (OuterVolumeSpecName: "dshm") pod "8e66f2aa-ba7f-4a24-b744-cf815dcca7ae" (UID: "8e66f2aa-ba7f-4a24-b744-cf815dcca7ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:07:14.120199 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.120109    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-kube-api-access-k87vg" (OuterVolumeSpecName: "kube-api-access-k87vg") pod "8e66f2aa-ba7f-4a24-b744-cf815dcca7ae" (UID: "8e66f2aa-ba7f-4a24-b744-cf815dcca7ae"). InnerVolumeSpecName "kube-api-access-k87vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:07:14.120199 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.120147    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e66f2aa-ba7f-4a24-b744-cf815dcca7ae" (UID: "8e66f2aa-ba7f-4a24-b744-cf815dcca7ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 03:07:14.219090 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.219058    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-tls-certs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:14.219090 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.219083    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-home\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:14.219090 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.219092    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k87vg\" (UniqueName: \"kubernetes.io/projected/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-kube-api-access-k87vg\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:14.219090 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.219100    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae-dshm\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:14.592747 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.592668    2565 generic.go:356] "Generic (PLEG): container finished" podID="8e66f2aa-ba7f-4a24-b744-cf815dcca7ae" containerID="f8dd40be17485ae4543f949d794e3125611e72ded486d078dab0bacecdac5ec8" exitCode=0
Jul 22 03:07:14.592747 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.592711    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"
Jul 22 03:07:14.592973 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.592746    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l" event={"ID":"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae","Type":"ContainerDied","Data":"f8dd40be17485ae4543f949d794e3125611e72ded486d078dab0bacecdac5ec8"}
Jul 22 03:07:14.592973 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.592788    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l" event={"ID":"8e66f2aa-ba7f-4a24-b744-cf815dcca7ae","Type":"ContainerDied","Data":"e54d09f02e7e01afcf3fc7e96d1ba5c4652491181c9991382530313feafcca10"}
Jul 22 03:07:14.592973 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.592813    2565 scope.go:117] "RemoveContainer" containerID="f8dd40be17485ae4543f949d794e3125611e72ded486d078dab0bacecdac5ec8"
Jul 22 03:07:14.594487 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.594464    2565 generic.go:356] "Generic (PLEG): container finished" podID="73293410-2066-48cb-9247-c849a0b1f70e" containerID="cc786bffd38cb5224ec8c52b206c329ae78f210621e30ff949ecd6dbe76be711" exitCode=0
Jul 22 03:07:14.594611 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.594498    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7730-140712170338320" event={"ID":"73293410-2066-48cb-9247-c849a0b1f70e","Type":"ContainerDied","Data":"cc786bffd38cb5224ec8c52b206c329ae78f210621e30ff949ecd6dbe76be711"}
Jul 22 03:07:14.601561 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.601546    2565 scope.go:117] "RemoveContainer" containerID="f8dd40be17485ae4543f949d794e3125611e72ded486d078dab0bacecdac5ec8"
Jul 22 03:07:14.601837 ip-10-0-140-200 kubenswrapper[2565]: E0722 03:07:14.601819    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8dd40be17485ae4543f949d794e3125611e72ded486d078dab0bacecdac5ec8\": container with ID starting with f8dd40be17485ae4543f949d794e3125611e72ded486d078dab0bacecdac5ec8 not found: ID does not exist" containerID="f8dd40be17485ae4543f949d794e3125611e72ded486d078dab0bacecdac5ec8"
Jul 22 03:07:14.601909 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.601866    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8dd40be17485ae4543f949d794e3125611e72ded486d078dab0bacecdac5ec8"} err="failed to get container status \"f8dd40be17485ae4543f949d794e3125611e72ded486d078dab0bacecdac5ec8\": rpc error: code = NotFound desc = could not find container \"f8dd40be17485ae4543f949d794e3125611e72ded486d078dab0bacecdac5ec8\": container with ID starting with f8dd40be17485ae4543f949d794e3125611e72ded486d078dab0bacecdac5ec8 not found: ID does not exist"
Jul 22 03:07:14.622561 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.622533    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"]
Jul 22 03:07:14.629000 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:14.628979    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d8076c/rate-test-model-d8076c-kserve-5965b6b48c-2th7l"]
Jul 22 03:07:15.726485 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:15.726457    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e66f2aa-ba7f-4a24-b744-cf815dcca7ae" path="/var/lib/kubelet/pods/8e66f2aa-ba7f-4a24-b744-cf815dcca7ae/volumes"
Jul 22 03:07:15.740157 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:15.740135    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7730-140712170338320"
Jul 22 03:07:15.759164 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:15.759139    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7730-140712170338320_73293410-2066-48cb-9247-c849a0b1f70e/test-curl-7730-140712170338320/0.log"
Jul 22 03:07:15.781515 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:15.781478    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7730-140712170338320"]
Jul 22 03:07:15.783550 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:15.783527    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7730-140712170338320"]
Jul 22 03:07:15.831963 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:15.831864    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snpmw\" (UniqueName: \"kubernetes.io/projected/73293410-2066-48cb-9247-c849a0b1f70e-kube-api-access-snpmw\") pod \"73293410-2066-48cb-9247-c849a0b1f70e\" (UID: \"73293410-2066-48cb-9247-c849a0b1f70e\") "
Jul 22 03:07:15.833789 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:15.833766    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73293410-2066-48cb-9247-c849a0b1f70e-kube-api-access-snpmw" (OuterVolumeSpecName: "kube-api-access-snpmw") pod "73293410-2066-48cb-9247-c849a0b1f70e" (UID: "73293410-2066-48cb-9247-c849a0b1f70e"). InnerVolumeSpecName "kube-api-access-snpmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:07:15.933165 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:15.933132    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snpmw\" (UniqueName: \"kubernetes.io/projected/73293410-2066-48cb-9247-c849a0b1f70e-kube-api-access-snpmw\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:16.606231 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:16.606197    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27ace1177869aeed056e1506f5f9c5ada4a7ed79e39b570b244d55d27522af95"
Jul 22 03:07:16.606231 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:16.606225    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7730-140712170338320"
Jul 22 03:07:16.615900 ip-10-0-140-200 kubenswrapper[2565]: E0722 03:07:16.615870    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7730-140712170338320\" is forbidden: User \"system:node:ip-10-0-140-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-200.ec2.internal' and this object" podUID="73293410-2066-48cb-9247-c849a0b1f70e" pod="opendatahub/test-curl-7730-140712170338320"
Jul 22 03:07:17.725944 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:17.725914    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73293410-2066-48cb-9247-c849a0b1f70e" path="/var/lib/kubelet/pods/73293410-2066-48cb-9247-c849a0b1f70e/volumes"
Jul 22 03:07:19.662038 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:19.661987    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:07:19.761120 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:19.761086    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:07:29.568956 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:29.568922    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"]
Jul 22 03:07:29.569443 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:29.569205    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75" podUID="801fcedf-8abe-4e56-be26-4d6adb7a8e28" containerName="maas-api" containerID="cri-o://f273a0934ee95ceaf6197fddfc72735623e8de1c87e07f624bfdbcf2c682ffb0" gracePeriod=30
Jul 22 03:07:29.611356 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:29.611328    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:07:29.824561 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:29.824497    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"
Jul 22 03:07:29.865568 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:29.865536    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/801fcedf-8abe-4e56-be26-4d6adb7a8e28-openshift-service-ca\") pod \"801fcedf-8abe-4e56-be26-4d6adb7a8e28\" (UID: \"801fcedf-8abe-4e56-be26-4d6adb7a8e28\") "
Jul 22 03:07:29.865746 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:29.865584    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxvtg\" (UniqueName: \"kubernetes.io/projected/801fcedf-8abe-4e56-be26-4d6adb7a8e28-kube-api-access-kxvtg\") pod \"801fcedf-8abe-4e56-be26-4d6adb7a8e28\" (UID: \"801fcedf-8abe-4e56-be26-4d6adb7a8e28\") "
Jul 22 03:07:29.865746 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:29.865647    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/801fcedf-8abe-4e56-be26-4d6adb7a8e28-maas-api-tls\") pod \"801fcedf-8abe-4e56-be26-4d6adb7a8e28\" (UID: \"801fcedf-8abe-4e56-be26-4d6adb7a8e28\") "
Jul 22 03:07:29.865987 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:29.865958    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801fcedf-8abe-4e56-be26-4d6adb7a8e28-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "801fcedf-8abe-4e56-be26-4d6adb7a8e28" (UID: "801fcedf-8abe-4e56-be26-4d6adb7a8e28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 03:07:29.867742 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:29.867718    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/801fcedf-8abe-4e56-be26-4d6adb7a8e28-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "801fcedf-8abe-4e56-be26-4d6adb7a8e28" (UID: "801fcedf-8abe-4e56-be26-4d6adb7a8e28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 03:07:29.867742 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:29.867732    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801fcedf-8abe-4e56-be26-4d6adb7a8e28-kube-api-access-kxvtg" (OuterVolumeSpecName: "kube-api-access-kxvtg") pod "801fcedf-8abe-4e56-be26-4d6adb7a8e28" (UID: "801fcedf-8abe-4e56-be26-4d6adb7a8e28"). InnerVolumeSpecName "kube-api-access-kxvtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:07:29.966367 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:29.966331    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/801fcedf-8abe-4e56-be26-4d6adb7a8e28-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:29.966367 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:29.966363    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxvtg\" (UniqueName: \"kubernetes.io/projected/801fcedf-8abe-4e56-be26-4d6adb7a8e28-kube-api-access-kxvtg\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:29.966367 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:29.966373    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/801fcedf-8abe-4e56-be26-4d6adb7a8e28-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:30.682322 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:30.682290    2565 generic.go:356] "Generic (PLEG): container finished" podID="801fcedf-8abe-4e56-be26-4d6adb7a8e28" containerID="f273a0934ee95ceaf6197fddfc72735623e8de1c87e07f624bfdbcf2c682ffb0" exitCode=0
Jul 22 03:07:30.682730 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:30.682328    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"
Jul 22 03:07:30.682730 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:30.682377    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75" event={"ID":"801fcedf-8abe-4e56-be26-4d6adb7a8e28","Type":"ContainerDied","Data":"f273a0934ee95ceaf6197fddfc72735623e8de1c87e07f624bfdbcf2c682ffb0"}
Jul 22 03:07:30.682730 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:30.682434    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75" event={"ID":"801fcedf-8abe-4e56-be26-4d6adb7a8e28","Type":"ContainerDied","Data":"a08d0a97c09ebb2275a984cea8ad62d01ef1e246c2505f1d549ce0be7509b4c5"}
Jul 22 03:07:30.682730 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:30.682462    2565 scope.go:117] "RemoveContainer" containerID="f273a0934ee95ceaf6197fddfc72735623e8de1c87e07f624bfdbcf2c682ffb0"
Jul 22 03:07:30.691064 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:30.691043    2565 scope.go:117] "RemoveContainer" containerID="f273a0934ee95ceaf6197fddfc72735623e8de1c87e07f624bfdbcf2c682ffb0"
Jul 22 03:07:30.691338 ip-10-0-140-200 kubenswrapper[2565]: E0722 03:07:30.691316    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f273a0934ee95ceaf6197fddfc72735623e8de1c87e07f624bfdbcf2c682ffb0\": container with ID starting with f273a0934ee95ceaf6197fddfc72735623e8de1c87e07f624bfdbcf2c682ffb0 not found: ID does not exist" containerID="f273a0934ee95ceaf6197fddfc72735623e8de1c87e07f624bfdbcf2c682ffb0"
Jul 22 03:07:30.691433 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:30.691344    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f273a0934ee95ceaf6197fddfc72735623e8de1c87e07f624bfdbcf2c682ffb0"} err="failed to get container status \"f273a0934ee95ceaf6197fddfc72735623e8de1c87e07f624bfdbcf2c682ffb0\": rpc error: code = NotFound desc = could not find container \"f273a0934ee95ceaf6197fddfc72735623e8de1c87e07f624bfdbcf2c682ffb0\": container with ID starting with f273a0934ee95ceaf6197fddfc72735623e8de1c87e07f624bfdbcf2c682ffb0 not found: ID does not exist"
Jul 22 03:07:30.703597 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:30.703576    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"]
Jul 22 03:07:30.714582 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:30.714557    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d8076c-66c6bd54df-lwc75"]
Jul 22 03:07:31.726507 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:31.726473    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="801fcedf-8abe-4e56-be26-4d6adb7a8e28" path="/var/lib/kubelet/pods/801fcedf-8abe-4e56-be26-4d6adb7a8e28/volumes"
Jul 22 03:07:34.097756 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:34.097725    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:07:34.525152 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:34.525124    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:07:40.506984 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:40.506951    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:07:40.608453 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:40.608421    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:07:40.717539 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:40.717494    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"]
Jul 22 03:07:40.717890 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:40.717835    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf" podUID="a225421d-45bf-4dcb-b18a-877fef022eeb" containerName="istio-proxy" containerID="cri-o://ba1eaec9b3620d534811919c01b4f06c8c228bfa8b72ebfa855bdf9b3a1040c2" gracePeriod=30
Jul 22 03:07:40.735960 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:40.735937    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:07:40.968653 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:40.968601    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:07:41.082194 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.082158    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cf82b3-4ghh5"]
Jul 22 03:07:41.082559 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.082544    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e66f2aa-ba7f-4a24-b744-cf815dcca7ae" containerName="main"
Jul 22 03:07:41.082559 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.082559    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e66f2aa-ba7f-4a24-b744-cf815dcca7ae" containerName="main"
Jul 22 03:07:41.082691 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.082590    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="801fcedf-8abe-4e56-be26-4d6adb7a8e28" containerName="maas-api"
Jul 22 03:07:41.082691 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.082596    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="801fcedf-8abe-4e56-be26-4d6adb7a8e28" containerName="maas-api"
Jul 22 03:07:41.082691 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.082602    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73293410-2066-48cb-9247-c849a0b1f70e" containerName="test-curl-7730-140712170338320"
Jul 22 03:07:41.082691 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.082610    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="73293410-2066-48cb-9247-c849a0b1f70e" containerName="test-curl-7730-140712170338320"
Jul 22 03:07:41.082691 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.082662    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="801fcedf-8abe-4e56-be26-4d6adb7a8e28" containerName="maas-api"
Jul 22 03:07:41.082691 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.082670    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e66f2aa-ba7f-4a24-b744-cf815dcca7ae" containerName="main"
Jul 22 03:07:41.082691 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.082684    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="73293410-2066-48cb-9247-c849a0b1f70e" containerName="test-curl-7730-140712170338320"
Jul 22 03:07:41.087878 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.087833    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cf82b3-4ghh5"
Jul 22 03:07:41.090472 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.090449    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vtlqd\""
Jul 22 03:07:41.092456 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.092435    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cf82b3-4ghh5"]
Jul 22 03:07:41.150263 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.150227    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51840344-a515-4520-9c57-f47c160ede4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-cf82b3-4ghh5\" (UID: \"51840344-a515-4520-9c57-f47c160ede4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cf82b3-4ghh5"
Jul 22 03:07:41.251678 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.251598    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51840344-a515-4520-9c57-f47c160ede4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-cf82b3-4ghh5\" (UID: \"51840344-a515-4520-9c57-f47c160ede4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cf82b3-4ghh5"
Jul 22 03:07:41.252210 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.252191    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51840344-a515-4520-9c57-f47c160ede4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-cf82b3-4ghh5\" (UID: \"51840344-a515-4520-9c57-f47c160ede4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cf82b3-4ghh5"
Jul 22 03:07:41.399968 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.399930    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cf82b3-4ghh5"
Jul 22 03:07:41.534046 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.533961    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cf82b3-4ghh5"]
Jul 22 03:07:41.537459 ip-10-0-140-200 kubenswrapper[2565]: W0722 03:07:41.537430    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51840344_a515_4520_9c57_f47c160ede4d.slice/crio-3fcd7eddde03d2ef60ffe3035996b8be6712aa2d2586d4fb28e7b2cc4d571ec6 WatchSource:0}: Error finding container 3fcd7eddde03d2ef60ffe3035996b8be6712aa2d2586d4fb28e7b2cc4d571ec6: Status 404 returned error can't find the container with id 3fcd7eddde03d2ef60ffe3035996b8be6712aa2d2586d4fb28e7b2cc4d571ec6
Jul 22 03:07:41.662256 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.662222    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:07:41.737217 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.737187    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cf82b3-4ghh5" event={"ID":"51840344-a515-4520-9c57-f47c160ede4d","Type":"ContainerStarted","Data":"3fcd7eddde03d2ef60ffe3035996b8be6712aa2d2586d4fb28e7b2cc4d571ec6"}
Jul 22 03:07:41.763253 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:41.763222    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:07:42.743291 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:42.743258    2565 generic.go:356] "Generic (PLEG): container finished" podID="51840344-a515-4520-9c57-f47c160ede4d" containerID="8f0f12732a5302c5edef9fd8415752ee3e385e25d4478a2c7ce9145c69e1e8e5" exitCode=0
Jul 22 03:07:42.743677 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:42.743314    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cf82b3-4ghh5" event={"ID":"51840344-a515-4520-9c57-f47c160ede4d","Type":"ContainerDied","Data":"8f0f12732a5302c5edef9fd8415752ee3e385e25d4478a2c7ce9145c69e1e8e5"}
Jul 22 03:07:43.733382 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:43.733346    2565 patch_prober.go:28] interesting pod/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 22 03:07:43.733558 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:43.733413    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf" podUID="a225421d-45bf-4dcb-b18a-877fef022eeb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 22 03:07:43.879362 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:43.879339    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cf82b3-4ghh5"
Jul 22 03:07:43.975504 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:43.975476    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51840344-a515-4520-9c57-f47c160ede4d-maas-api-service-ca\") pod \"51840344-a515-4520-9c57-f47c160ede4d\" (UID: \"51840344-a515-4520-9c57-f47c160ede4d\") "
Jul 22 03:07:43.975813 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:43.975792    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51840344-a515-4520-9c57-f47c160ede4d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "51840344-a515-4520-9c57-f47c160ede4d" (UID: "51840344-a515-4520-9c57-f47c160ede4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 03:07:44.076239 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:44.076168    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51840344-a515-4520-9c57-f47c160ede4d-maas-api-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:44.753983 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:44.753947    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cf82b3-4ghh5"
Jul 22 03:07:44.754134 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:44.753979    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cf82b3-4ghh5" event={"ID":"51840344-a515-4520-9c57-f47c160ede4d","Type":"ContainerDied","Data":"3fcd7eddde03d2ef60ffe3035996b8be6712aa2d2586d4fb28e7b2cc4d571ec6"}
Jul 22 03:07:44.754134 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:44.754046    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fcd7eddde03d2ef60ffe3035996b8be6712aa2d2586d4fb28e7b2cc4d571ec6"
Jul 22 03:07:45.965730 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.965700    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:07:45.995178 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.995109    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-token\") pod \"a225421d-45bf-4dcb-b18a-877fef022eeb\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") "
Jul 22 03:07:45.995178 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.995159    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-credential-socket\") pod \"a225421d-45bf-4dcb-b18a-877fef022eeb\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") "
Jul 22 03:07:45.995381 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.995185    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-workload-certs\") pod \"a225421d-45bf-4dcb-b18a-877fef022eeb\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") "
Jul 22 03:07:45.995381 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.995216    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-podinfo\") pod \"a225421d-45bf-4dcb-b18a-877fef022eeb\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") "
Jul 22 03:07:45.995381 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.995237    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a225421d-45bf-4dcb-b18a-877fef022eeb-istiod-ca-cert\") pod \"a225421d-45bf-4dcb-b18a-877fef022eeb\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") "
Jul 22 03:07:45.995381 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.995317    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-envoy\") pod \"a225421d-45bf-4dcb-b18a-877fef022eeb\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") "
Jul 22 03:07:45.995381 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.995371    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-data\") pod \"a225421d-45bf-4dcb-b18a-877fef022eeb\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") "
Jul 22 03:07:45.995656 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.995398    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktbqw\" (UniqueName: \"kubernetes.io/projected/a225421d-45bf-4dcb-b18a-877fef022eeb-kube-api-access-ktbqw\") pod \"a225421d-45bf-4dcb-b18a-877fef022eeb\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") "
Jul 22 03:07:45.995656 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.995424    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-workload-socket\") pod \"a225421d-45bf-4dcb-b18a-877fef022eeb\" (UID: \"a225421d-45bf-4dcb-b18a-877fef022eeb\") "
Jul 22 03:07:45.995656 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.995471    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a225421d-45bf-4dcb-b18a-877fef022eeb" (UID: "a225421d-45bf-4dcb-b18a-877fef022eeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:07:45.995656 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.995488    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a225421d-45bf-4dcb-b18a-877fef022eeb" (UID: "a225421d-45bf-4dcb-b18a-877fef022eeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:07:45.995888 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.995715    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-workload-certs\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:45.995888 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.995733    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-credential-socket\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:45.995888 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.995809    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a225421d-45bf-4dcb-b18a-877fef022eeb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a225421d-45bf-4dcb-b18a-877fef022eeb" (UID: "a225421d-45bf-4dcb-b18a-877fef022eeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 03:07:45.996042 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.995994    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a225421d-45bf-4dcb-b18a-877fef022eeb" (UID: "a225421d-45bf-4dcb-b18a-877fef022eeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:07:45.996356 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.996325    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-data" (OuterVolumeSpecName: "istio-data") pod "a225421d-45bf-4dcb-b18a-877fef022eeb" (UID: "a225421d-45bf-4dcb-b18a-877fef022eeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:07:45.997481 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.997454    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-token" (OuterVolumeSpecName: "istio-token") pod "a225421d-45bf-4dcb-b18a-877fef022eeb" (UID: "a225421d-45bf-4dcb-b18a-877fef022eeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:07:45.997585 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.997474    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a225421d-45bf-4dcb-b18a-877fef022eeb" (UID: "a225421d-45bf-4dcb-b18a-877fef022eeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 22 03:07:45.997734 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.997718    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a225421d-45bf-4dcb-b18a-877fef022eeb-kube-api-access-ktbqw" (OuterVolumeSpecName: "kube-api-access-ktbqw") pod "a225421d-45bf-4dcb-b18a-877fef022eeb" (UID: "a225421d-45bf-4dcb-b18a-877fef022eeb"). InnerVolumeSpecName "kube-api-access-ktbqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:07:45.997893 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:45.997872    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a225421d-45bf-4dcb-b18a-877fef022eeb" (UID: "a225421d-45bf-4dcb-b18a-877fef022eeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:07:46.096988 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:46.096957    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-data\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:46.096988 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:46.096983    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktbqw\" (UniqueName: \"kubernetes.io/projected/a225421d-45bf-4dcb-b18a-877fef022eeb-kube-api-access-ktbqw\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:46.097163 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:46.096998    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-workload-socket\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:46.097163 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:46.097009    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-token\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:46.097163 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:46.097021    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-podinfo\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:46.097163 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:46.097031    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a225421d-45bf-4dcb-b18a-877fef022eeb-istiod-ca-cert\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:46.097163 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:46.097042    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a225421d-45bf-4dcb-b18a-877fef022eeb-istio-envoy\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:46.764809 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:46.764776    2565 generic.go:356] "Generic (PLEG): container finished" podID="a225421d-45bf-4dcb-b18a-877fef022eeb" containerID="ba1eaec9b3620d534811919c01b4f06c8c228bfa8b72ebfa855bdf9b3a1040c2" exitCode=0
Jul 22 03:07:46.764990 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:46.764828    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"
Jul 22 03:07:46.764990 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:46.764878    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf" event={"ID":"a225421d-45bf-4dcb-b18a-877fef022eeb","Type":"ContainerDied","Data":"ba1eaec9b3620d534811919c01b4f06c8c228bfa8b72ebfa855bdf9b3a1040c2"}
Jul 22 03:07:46.764990 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:46.764927    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf" event={"ID":"a225421d-45bf-4dcb-b18a-877fef022eeb","Type":"ContainerDied","Data":"f1f6ad8b693e9d374d3907313ab0aa7cdfb7b43a841395f79439ed3368c688d6"}
Jul 22 03:07:46.764990 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:46.764949    2565 scope.go:117] "RemoveContainer" containerID="ba1eaec9b3620d534811919c01b4f06c8c228bfa8b72ebfa855bdf9b3a1040c2"
Jul 22 03:07:46.774937 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:46.774919    2565 scope.go:117] "RemoveContainer" containerID="e2bede040c9f889092783ca8d4217b6fd116b70aec7a49e0bd196aa77e3b7474"
Jul 22 03:07:46.782879 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:46.782844    2565 scope.go:117] "RemoveContainer" containerID="ba1eaec9b3620d534811919c01b4f06c8c228bfa8b72ebfa855bdf9b3a1040c2"
Jul 22 03:07:46.783106 ip-10-0-140-200 kubenswrapper[2565]: E0722 03:07:46.783090    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba1eaec9b3620d534811919c01b4f06c8c228bfa8b72ebfa855bdf9b3a1040c2\": container with ID starting with ba1eaec9b3620d534811919c01b4f06c8c228bfa8b72ebfa855bdf9b3a1040c2 not found: ID does not exist" containerID="ba1eaec9b3620d534811919c01b4f06c8c228bfa8b72ebfa855bdf9b3a1040c2"
Jul 22 03:07:46.783155 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:46.783114    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba1eaec9b3620d534811919c01b4f06c8c228bfa8b72ebfa855bdf9b3a1040c2"} err="failed to get container status \"ba1eaec9b3620d534811919c01b4f06c8c228bfa8b72ebfa855bdf9b3a1040c2\": rpc error: code = NotFound desc = could not find container \"ba1eaec9b3620d534811919c01b4f06c8c228bfa8b72ebfa855bdf9b3a1040c2\": container with ID starting with ba1eaec9b3620d534811919c01b4f06c8c228bfa8b72ebfa855bdf9b3a1040c2 not found: ID does not exist"
Jul 22 03:07:46.783155 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:46.783132    2565 scope.go:117] "RemoveContainer" containerID="e2bede040c9f889092783ca8d4217b6fd116b70aec7a49e0bd196aa77e3b7474"
Jul 22 03:07:46.783347 ip-10-0-140-200 kubenswrapper[2565]: E0722 03:07:46.783326    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2bede040c9f889092783ca8d4217b6fd116b70aec7a49e0bd196aa77e3b7474\": container with ID starting with e2bede040c9f889092783ca8d4217b6fd116b70aec7a49e0bd196aa77e3b7474 not found: ID does not exist" containerID="e2bede040c9f889092783ca8d4217b6fd116b70aec7a49e0bd196aa77e3b7474"
Jul 22 03:07:46.783408 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:46.783345    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2bede040c9f889092783ca8d4217b6fd116b70aec7a49e0bd196aa77e3b7474"} err="failed to get container status \"e2bede040c9f889092783ca8d4217b6fd116b70aec7a49e0bd196aa77e3b7474\": rpc error: code = NotFound desc = could not find container \"e2bede040c9f889092783ca8d4217b6fd116b70aec7a49e0bd196aa77e3b7474\": container with ID starting with e2bede040c9f889092783ca8d4217b6fd116b70aec7a49e0bd196aa77e3b7474 not found: ID does not exist"
Jul 22 03:07:46.786398 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:46.786373    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"]
Jul 22 03:07:46.792497 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:46.792478    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-d8076c-openshift-default-f44c88779-xxscf"]
Jul 22 03:07:47.726200 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:47.726171    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a225421d-45bf-4dcb-b18a-877fef022eeb" path="/var/lib/kubelet/pods/a225421d-45bf-4dcb-b18a-877fef022eeb/volumes"
Jul 22 03:07:51.585033 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:51.584997    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"]
Jul 22 03:07:51.585539 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:51.585222    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl" podUID="58bb7202-f9fb-41a2-adfe-4d007f52e3fb" containerName="maas-api" containerID="cri-o://c04ed884d5e97fc69c6200035141e522eb7f185bd12438b30336771d00e7c438" gracePeriod=30
Jul 22 03:07:51.614371 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:51.614343    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:07:51.790570 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:51.790539    2565 generic.go:356] "Generic (PLEG): container finished" podID="58bb7202-f9fb-41a2-adfe-4d007f52e3fb" containerID="c04ed884d5e97fc69c6200035141e522eb7f185bd12438b30336771d00e7c438" exitCode=0
Jul 22 03:07:51.790743 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:51.790625    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl" event={"ID":"58bb7202-f9fb-41a2-adfe-4d007f52e3fb","Type":"ContainerDied","Data":"c04ed884d5e97fc69c6200035141e522eb7f185bd12438b30336771d00e7c438"}
Jul 22 03:07:51.836024 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:51.835966    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"
Jul 22 03:07:51.945187 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:51.945161    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xv76\" (UniqueName: \"kubernetes.io/projected/58bb7202-f9fb-41a2-adfe-4d007f52e3fb-kube-api-access-5xv76\") pod \"58bb7202-f9fb-41a2-adfe-4d007f52e3fb\" (UID: \"58bb7202-f9fb-41a2-adfe-4d007f52e3fb\") "
Jul 22 03:07:51.945347 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:51.945213    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58bb7202-f9fb-41a2-adfe-4d007f52e3fb-maas-api-tls\") pod \"58bb7202-f9fb-41a2-adfe-4d007f52e3fb\" (UID: \"58bb7202-f9fb-41a2-adfe-4d007f52e3fb\") "
Jul 22 03:07:51.945347 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:51.945245    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58bb7202-f9fb-41a2-adfe-4d007f52e3fb-openshift-service-ca\") pod \"58bb7202-f9fb-41a2-adfe-4d007f52e3fb\" (UID: \"58bb7202-f9fb-41a2-adfe-4d007f52e3fb\") "
Jul 22 03:07:51.945612 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:51.945591    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58bb7202-f9fb-41a2-adfe-4d007f52e3fb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "58bb7202-f9fb-41a2-adfe-4d007f52e3fb" (UID: "58bb7202-f9fb-41a2-adfe-4d007f52e3fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 03:07:51.947146 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:51.947116    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58bb7202-f9fb-41a2-adfe-4d007f52e3fb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "58bb7202-f9fb-41a2-adfe-4d007f52e3fb" (UID: "58bb7202-f9fb-41a2-adfe-4d007f52e3fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 03:07:51.947244 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:51.947216    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58bb7202-f9fb-41a2-adfe-4d007f52e3fb-kube-api-access-5xv76" (OuterVolumeSpecName: "kube-api-access-5xv76") pod "58bb7202-f9fb-41a2-adfe-4d007f52e3fb" (UID: "58bb7202-f9fb-41a2-adfe-4d007f52e3fb"). InnerVolumeSpecName "kube-api-access-5xv76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:07:52.045955 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:52.045928    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xv76\" (UniqueName: \"kubernetes.io/projected/58bb7202-f9fb-41a2-adfe-4d007f52e3fb-kube-api-access-5xv76\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:52.045955 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:52.045955    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58bb7202-f9fb-41a2-adfe-4d007f52e3fb-maas-api-tls\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:52.046134 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:52.045969    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58bb7202-f9fb-41a2-adfe-4d007f52e3fb-openshift-service-ca\") on node \"ip-10-0-140-200.ec2.internal\" DevicePath \"\""
Jul 22 03:07:52.796367 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:52.796333    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"
Jul 22 03:07:52.796877 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:52.796382    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl" event={"ID":"58bb7202-f9fb-41a2-adfe-4d007f52e3fb","Type":"ContainerDied","Data":"4d907f463937c2213a9c53cc4ff201481daf8466e3a4c4b9055f2d9172b91b9a"}
Jul 22 03:07:52.796877 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:52.796450    2565 scope.go:117] "RemoveContainer" containerID="c04ed884d5e97fc69c6200035141e522eb7f185bd12438b30336771d00e7c438"
Jul 22 03:07:52.817709 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:52.817677    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"]
Jul 22 03:07:52.821252 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:52.821230    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cf82b3-6b4bf788-dpkgl"]
Jul 22 03:07:53.726447 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:53.726417    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58bb7202-f9fb-41a2-adfe-4d007f52e3fb" path="/var/lib/kubelet/pods/58bb7202-f9fb-41a2-adfe-4d007f52e3fb/volumes"
Jul 22 03:07:56.162776 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:56.162744    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:07:56.166962 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:56.166935    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cf82b3-4ghh5"]
Jul 22 03:07:56.169313 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:56.169292    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cf82b3-4ghh5"]
Jul 22 03:07:56.557983 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:56.557904    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:07:57.725994 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:07:57.725960    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51840344-a515-4520-9c57-f47c160ede4d" path="/var/lib/kubelet/pods/51840344-a515-4520-9c57-f47c160ede4d/volumes"
Jul 22 03:08:02.524804 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:02.524767    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:08:02.662914 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:02.662879    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:08:02.816449 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:02.816364    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:08:03.048839 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:03.048809    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:08:08.878982 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:08.878952    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-flrgk_9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec/manager/0.log"
Jul 22 03:08:09.002047 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:09.002018    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-fpw79_9541f8fd-025b-4524-873f-7fc567fcc58a/manager/0.log"
Jul 22 03:08:09.126449 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:09.126416    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b897bdc49-pm7x6_fafd6279-1ccd-4c2e-b5ee-88876154879b/manager/0.log"
Jul 22 03:08:09.250012 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:09.249934    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ckcnw_a1fe2929-e2c8-4231-ba3d-547a7b982853/server/0.log"
Jul 22 03:08:09.368717 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:09.368689    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qhj4q_77956685-f072-4905-a803-ebef81ed3145/manager/0.log"
Jul 22 03:08:09.593748 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:09.593673    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:08:09.764450 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:09.764423    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fcd77b786-zzhbg_27e667ef-ad44-4c46-9a62-7c34985a6713/copy-manifests/0.log"
Jul 22 03:08:09.776885 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:09.776858    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fcd77b786-zzhbg_27e667ef-ad44-4c46-9a62-7c34985a6713/manager/0.log"
Jul 22 03:08:10.132862 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:10.132817    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-76d9db859c-rhpqn_e393c1da-9ce4-4cf0-91a1-66ca299217d1/maas-api/0.log"
Jul 22 03:08:10.360446 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:10.360417    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744820-dvcgf_64f7a59a-f6ab-46f0-9da7-8b5a5f66ef47/cleanup/0.log"
Jul 22 03:08:10.471869 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:10.471766    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-njx2t_5caa01c0-91bb-4b72-877e-27e01d39395c/postgres/0.log"
Jul 22 03:08:11.905962 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:11.905929    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-w7vvk_23999d7a-c2c0-421a-be84-13d16b314726/manager/0.log"
Jul 22 03:08:12.252883 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:12.252796    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-hrkpx_b131391a-810c-4f76-be3e-4dfe4decba2c/registry-server/0.log"
Jul 22 03:08:12.619860 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:12.619816    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-grkxg_a627716e-c9d3-4b38-9494-1aebebcd1882/manager/0.log"
Jul 22 03:08:12.981110 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:12.981038    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4ck5mw_aaae1329-63c3-460e-afcd-114656c1f9e1/istio-proxy/0.log"
Jul 22 03:08:13.230648 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:13.230622    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79676bf5cc-fgwxm_5d7f7d87-3443-42f1-b8c6-c07d6f272922/kube-auth-proxy/0.log"
Jul 22 03:08:13.470700 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:13.470664    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xw2bp_3b40b006-204b-4291-901e-af512339c609/istio-proxy/1.log"
Jul 22 03:08:13.714785 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:13.714752    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-vdmlh_84cfc958-d425-471a-ab6f-cfcd0d8bc121/payload-processing/0.log"
Jul 22 03:08:14.394493 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:14.394464    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-pvmpd_0e59b6f1-67b6-4bc1-8535-df5ca681b7e1/main/0.log"
Jul 22 03:08:14.613318 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:14.613290    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-k9zkf_c9bfbdeb-f030-48d0-a8a4-4f660c937c48/main/0.log"
Jul 22 03:08:20.050181 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.050149    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h4tn6/must-gather-nnhc5"]
Jul 22 03:08:20.050656 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.050530    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58bb7202-f9fb-41a2-adfe-4d007f52e3fb" containerName="maas-api"
Jul 22 03:08:20.050656 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.050541    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="58bb7202-f9fb-41a2-adfe-4d007f52e3fb" containerName="maas-api"
Jul 22 03:08:20.050656 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.050557    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51840344-a515-4520-9c57-f47c160ede4d" containerName="revoke-keys"
Jul 22 03:08:20.050656 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.050564    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="51840344-a515-4520-9c57-f47c160ede4d" containerName="revoke-keys"
Jul 22 03:08:20.050656 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.050573    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a225421d-45bf-4dcb-b18a-877fef022eeb" containerName="istio-proxy"
Jul 22 03:08:20.050656 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.050578    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a225421d-45bf-4dcb-b18a-877fef022eeb" containerName="istio-proxy"
Jul 22 03:08:20.050656 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.050589    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a225421d-45bf-4dcb-b18a-877fef022eeb" containerName="istio-proxy"
Jul 22 03:08:20.050656 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.050594    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a225421d-45bf-4dcb-b18a-877fef022eeb" containerName="istio-proxy"
Jul 22 03:08:20.050656 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.050657    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="58bb7202-f9fb-41a2-adfe-4d007f52e3fb" containerName="maas-api"
Jul 22 03:08:20.050997 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.050669    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a225421d-45bf-4dcb-b18a-877fef022eeb" containerName="istio-proxy"
Jul 22 03:08:20.050997 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.050678    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="51840344-a515-4520-9c57-f47c160ede4d" containerName="revoke-keys"
Jul 22 03:08:20.050997 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.050783    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a225421d-45bf-4dcb-b18a-877fef022eeb" containerName="istio-proxy"
Jul 22 03:08:20.054094 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.054072    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h4tn6/must-gather-nnhc5"
Jul 22 03:08:20.056857 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.056818    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h4tn6\"/\"openshift-service-ca.crt\""
Jul 22 03:08:20.056980 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.056863    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h4tn6\"/\"kube-root-ca.crt\""
Jul 22 03:08:20.057763 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.057748    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-h4tn6\"/\"default-dockercfg-cr7fc\""
Jul 22 03:08:20.064347 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.064317    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:08:20.064768 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.064748    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h4tn6/must-gather-nnhc5"]
Jul 22 03:08:20.149428 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.149399    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log" limit=1024
Jul 22 03:08:20.181687 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.181656    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m2lp\" (UniqueName: \"kubernetes.io/projected/7f6a0585-3213-4e78-9c23-0ee08b751ebf-kube-api-access-6m2lp\") pod \"must-gather-nnhc5\" (UID: \"7f6a0585-3213-4e78-9c23-0ee08b751ebf\") " pod="openshift-must-gather-h4tn6/must-gather-nnhc5"
Jul 22 03:08:20.181823 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.181709    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7f6a0585-3213-4e78-9c23-0ee08b751ebf-must-gather-output\") pod \"must-gather-nnhc5\" (UID: \"7f6a0585-3213-4e78-9c23-0ee08b751ebf\") " pod="openshift-must-gather-h4tn6/must-gather-nnhc5"
Jul 22 03:08:20.282960 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.282927    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m2lp\" (UniqueName: \"kubernetes.io/projected/7f6a0585-3213-4e78-9c23-0ee08b751ebf-kube-api-access-6m2lp\") pod \"must-gather-nnhc5\" (UID: \"7f6a0585-3213-4e78-9c23-0ee08b751ebf\") " pod="openshift-must-gather-h4tn6/must-gather-nnhc5"
Jul 22 03:08:20.283119 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.282970    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7f6a0585-3213-4e78-9c23-0ee08b751ebf-must-gather-output\") pod \"must-gather-nnhc5\" (UID: \"7f6a0585-3213-4e78-9c23-0ee08b751ebf\") " pod="openshift-must-gather-h4tn6/must-gather-nnhc5"
Jul 22 03:08:20.283271 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.283255    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7f6a0585-3213-4e78-9c23-0ee08b751ebf-must-gather-output\") pod \"must-gather-nnhc5\" (UID: \"7f6a0585-3213-4e78-9c23-0ee08b751ebf\") " pod="openshift-must-gather-h4tn6/must-gather-nnhc5"
Jul 22 03:08:20.291780 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.291760    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m2lp\" (UniqueName: \"kubernetes.io/projected/7f6a0585-3213-4e78-9c23-0ee08b751ebf-kube-api-access-6m2lp\") pod \"must-gather-nnhc5\" (UID: \"7f6a0585-3213-4e78-9c23-0ee08b751ebf\") " pod="openshift-must-gather-h4tn6/must-gather-nnhc5"
Jul 22 03:08:20.363345 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.363320    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h4tn6/must-gather-nnhc5"
Jul 22 03:08:20.498884 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.498840    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h4tn6/must-gather-nnhc5"]
Jul 22 03:08:20.501119 ip-10-0-140-200 kubenswrapper[2565]: W0722 03:08:20.501080    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f6a0585_3213_4e78_9c23_0ee08b751ebf.slice/crio-f02c4bc6ff5ea516bb80cd9a0fdf64308b256ac6b0824a83440f4ef0b39d7977 WatchSource:0}: Error finding container f02c4bc6ff5ea516bb80cd9a0fdf64308b256ac6b0824a83440f4ef0b39d7977: Status 404 returned error can't find the container with id f02c4bc6ff5ea516bb80cd9a0fdf64308b256ac6b0824a83440f4ef0b39d7977
Jul 22 03:08:20.926008 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:20.925976    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h4tn6/must-gather-nnhc5" event={"ID":"7f6a0585-3213-4e78-9c23-0ee08b751ebf","Type":"ContainerStarted","Data":"f02c4bc6ff5ea516bb80cd9a0fdf64308b256ac6b0824a83440f4ef0b39d7977"}
Jul 22 03:08:21.935201 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:21.935158    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h4tn6/must-gather-nnhc5" event={"ID":"7f6a0585-3213-4e78-9c23-0ee08b751ebf","Type":"ContainerStarted","Data":"7ffb31aa3b8f54e780a4bec462590406332dcb06a22a68f13c47e4aa48044887"}
Jul 22 03:08:21.936458 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:21.936425    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h4tn6/must-gather-nnhc5" event={"ID":"7f6a0585-3213-4e78-9c23-0ee08b751ebf","Type":"ContainerStarted","Data":"89bc9d7502581b42772ae18f7a9e81bc67b902785d7a36537c663b1f4c40ba56"}
Jul 22 03:08:21.957061 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:21.956992    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h4tn6/must-gather-nnhc5" podStartSLOduration=1.210627128 podStartE2EDuration="1.956971683s" totalImagesPullingTime="746.344555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:08:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:08:21.954131404 +0000 UTC m=+2852.802467972" watchObservedRunningTime="2026-07-22 03:08:21.956971683 +0000 UTC m=+2852.805308258"
Jul 22 03:08:22.951415 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:22.951374    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kzv8x_faebb5f3-7db6-4e1c-88b0-a4530442e728/global-pull-secret-syncer/0.log"
Jul 22 03:08:23.010891 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:23.010828    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4ggf4_d9824be5-cf4c-49b5-9ad4-db70c783d24c/konnectivity-agent/0.log"
Jul 22 03:08:23.190703 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:23.190671    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-200.ec2.internal_a0b30ae5bc27eeb039f1a3250327b5a6/haproxy/0.log"
Jul 22 03:08:27.826782 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:27.826745    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-w7vvk_23999d7a-c2c0-421a-be84-13d16b314726/manager/0.log"
Jul 22 03:08:27.961979 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:27.961949    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-hrkpx_b131391a-810c-4f76-be3e-4dfe4decba2c/registry-server/0.log"
Jul 22 03:08:28.316539 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:28.316507    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-grkxg_a627716e-c9d3-4b38-9494-1aebebcd1882/manager/0.log"
Jul 22 03:08:30.591158 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:30.591120    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t74nq_9dd103b8-b2d4-45fc-9194-9de212c9410f/node-exporter/0.log"
Jul 22 03:08:30.621208 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:30.621177    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t74nq_9dd103b8-b2d4-45fc-9194-9de212c9410f/kube-rbac-proxy/0.log"
Jul 22 03:08:30.653142 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:30.653111    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t74nq_9dd103b8-b2d4-45fc-9194-9de212c9410f/init-textfile/0.log"
Jul 22 03:08:33.044773 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:33.044740    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"]
Jul 22 03:08:33.051108 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:33.051081    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"
Jul 22 03:08:33.059024 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:33.058964    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"]
Jul 22 03:08:33.109512 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:33.109476    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb-sys\") pod \"perf-node-gather-daemonset-pf4tc\" (UID: \"f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb\") " pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"
Jul 22 03:08:33.109913 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:33.109888    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb-proc\") pod \"perf-node-gather-daemonset-pf4tc\" (UID: \"f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb\") " pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"
Jul 22 03:08:33.110140 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:33.110100    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-599rj\" (UniqueName: \"kubernetes.io/projected/f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb-kube-api-access-599rj\") pod \"perf-node-gather-daemonset-pf4tc\" (UID: \"f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb\") " pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"
Jul 22 03:08:33.110350 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:33.110304    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb-podres\") pod \"perf-node-gather-daemonset-pf4tc\" (UID: \"f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb\") " pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"
Jul 22 03:08:33.110511 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:33.110493    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb-lib-modules\") pod \"perf-node-gather-daemonset-pf4tc\" (UID: \"f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb\") " pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"
Jul 22 03:08:33.211245 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:33.211214    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb-proc\") pod \"perf-node-gather-daemonset-pf4tc\" (UID: \"f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb\") " pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"
Jul 22 03:08:33.211245 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:33.211258    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-599rj\" (UniqueName: \"kubernetes.io/projected/f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb-kube-api-access-599rj\") pod \"perf-node-gather-daemonset-pf4tc\" (UID: \"f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb\") " pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"
Jul 22 03:08:33.211470 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:33.211291    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb-podres\") pod \"perf-node-gather-daemonset-pf4tc\" (UID: \"f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb\") " pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"
Jul 22 03:08:33.211470 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:33.211311    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb-lib-modules\") pod \"perf-node-gather-daemonset-pf4tc\" (UID: \"f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb\") " pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"
Jul 22 03:08:33.211470 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:33.211356    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb-proc\") pod \"perf-node-gather-daemonset-pf4tc\" (UID: \"f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb\") " pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"
Jul 22 03:08:33.211470 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:33.211372    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb-sys\") pod \"perf-node-gather-daemonset-pf4tc\" (UID: \"f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb\") " pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"
Jul 22 03:08:33.211470 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:33.211430    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb-sys\") pod \"perf-node-gather-daemonset-pf4tc\" (UID: \"f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb\") " pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"
Jul 22 03:08:33.211701 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:33.211569    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb-podres\") pod \"perf-node-gather-daemonset-pf4tc\" (UID: \"f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb\") " pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"
Jul 22 03:08:33.211701 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:33.211593    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb-lib-modules\") pod \"perf-node-gather-daemonset-pf4tc\" (UID: \"f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb\") " pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"
Jul 22 03:08:33.219500 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:33.219469    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-599rj\" (UniqueName: \"kubernetes.io/projected/f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb-kube-api-access-599rj\") pod \"perf-node-gather-daemonset-pf4tc\" (UID: \"f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb\") " pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"
Jul 22 03:08:33.367945 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:33.367904    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"
Jul 22 03:08:33.562715 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:33.562690    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"]
Jul 22 03:08:33.568940 ip-10-0-140-200 kubenswrapper[2565]: W0722 03:08:33.568738    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf73332cd_e4d5_4b8c_82ed_5b7d3f5e40fb.slice/crio-3c83f44e03faf7869636543e005c8a5cff4f37fccf058803c65dab20e5322deb WatchSource:0}: Error finding container 3c83f44e03faf7869636543e005c8a5cff4f37fccf058803c65dab20e5322deb: Status 404 returned error can't find the container with id 3c83f44e03faf7869636543e005c8a5cff4f37fccf058803c65dab20e5322deb
Jul 22 03:08:34.022620 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:34.022525    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc" event={"ID":"f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb","Type":"ContainerStarted","Data":"8005ba7cb7533371da1cb1fc0d0112395c27d6c08fcaccb8f3b8ad768434e7de"}
Jul 22 03:08:34.022620 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:34.022590    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"
Jul 22 03:08:34.022620 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:34.022608    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc" event={"ID":"f73332cd-e4d5-4b8c-82ed-5b7d3f5e40fb","Type":"ContainerStarted","Data":"3c83f44e03faf7869636543e005c8a5cff4f37fccf058803c65dab20e5322deb"}
Jul 22 03:08:34.040941 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:34.040888    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc" podStartSLOduration=1.040873779 podStartE2EDuration="1.040873779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 03:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:08:34.040668512 +0000 UTC m=+2864.889005085" watchObservedRunningTime="2026-07-22 03:08:34.040873779 +0000 UTC m=+2864.889210352"
Jul 22 03:08:35.315274 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:35.315240    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k2gpv_417b06e9-0b5a-486a-92fa-aa9e118b7c36/dns/0.log"
Jul 22 03:08:35.335798 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:35.335768    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k2gpv_417b06e9-0b5a-486a-92fa-aa9e118b7c36/kube-rbac-proxy/0.log"
Jul 22 03:08:35.406074 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:35.406047    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zhtxk_f5be685a-0b0f-4058-9272-2fb16817f84d/dns-node-resolver/0.log"
Jul 22 03:08:35.942097 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:35.942062    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7cff98f456-jkdf8_1e6149cf-f721-48fa-9255-04dcb0486502/registry/0.log"
Jul 22 03:08:35.984835 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:35.984803    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v6sck_a1a137a5-3d05-45cf-8620-7537635e87d0/node-ca/0.log"
Jul 22 03:08:36.901563 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:36.901529    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4ck5mw_aaae1329-63c3-460e-afcd-114656c1f9e1/istio-proxy/0.log"
Jul 22 03:08:37.146708 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:37.146678    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79676bf5cc-fgwxm_5d7f7d87-3443-42f1-b8c6-c07d6f272922/kube-auth-proxy/0.log"
Jul 22 03:08:37.198966 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:37.198889    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xw2bp_3b40b006-204b-4291-901e-af512339c609/istio-proxy/0.log"
Jul 22 03:08:37.232734 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:37.232701    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xw2bp_3b40b006-204b-4291-901e-af512339c609/istio-proxy/1.log"
Jul 22 03:08:37.326960 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:37.326935    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-vdmlh_84cfc958-d425-471a-ab6f-cfcd0d8bc121/payload-processing/0.log"
Jul 22 03:08:38.087106 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:38.087079    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p6j8z_cfb4b1be-f309-43a2-8a40-b109ef938765/serve-healthcheck-canary/0.log"
Jul 22 03:08:38.558527 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:38.558499    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7984v_50921dd8-5d4d-4801-a4c5-339f81901b9f/kube-rbac-proxy/0.log"
Jul 22 03:08:38.577865 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:38.577828    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7984v_50921dd8-5d4d-4801-a4c5-339f81901b9f/exporter/0.log"
Jul 22 03:08:38.600101 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:38.600073    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7984v_50921dd8-5d4d-4801-a4c5-339f81901b9f/extractor/0.log"
Jul 22 03:08:40.041100 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:40.041071    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-h4tn6/perf-node-gather-daemonset-pf4tc"
Jul 22 03:08:40.837730 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:40.837707    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-flrgk_9b22e1aa-35d2-43f2-b9a0-60fa2546d1ec/manager/0.log"
Jul 22 03:08:41.074239 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:41.074202    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-fpw79_9541f8fd-025b-4524-873f-7fc567fcc58a/manager/0.log"
Jul 22 03:08:41.221997 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:41.221968    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b897bdc49-pm7x6_fafd6279-1ccd-4c2e-b5ee-88876154879b/manager/0.log"
Jul 22 03:08:41.243605 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:41.243580    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ckcnw_a1fe2929-e2c8-4231-ba3d-547a7b982853/server/0.log"
Jul 22 03:08:41.340692 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:41.340664    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qhj4q_77956685-f072-4905-a803-ebef81ed3145/manager/0.log"
Jul 22 03:08:41.569710 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:41.569574    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fcd77b786-zzhbg_27e667ef-ad44-4c46-9a62-7c34985a6713/manager/0.log"
Jul 22 03:08:41.596563 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:41.596537    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fcd77b786-zzhbg_27e667ef-ad44-4c46-9a62-7c34985a6713/copy-manifests/0.log"
Jul 22 03:08:43.121047 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:43.121013    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4ck5mw_aaae1329-63c3-460e-afcd-114656c1f9e1/istio-proxy/0.log"
Jul 22 03:08:43.370998 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:43.370972    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79676bf5cc-fgwxm_5d7f7d87-3443-42f1-b8c6-c07d6f272922/kube-auth-proxy/0.log"
Jul 22 03:08:43.406065 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:43.405992    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xw2bp_3b40b006-204b-4291-901e-af512339c609/istio-proxy/0.log"
Jul 22 03:08:43.439490 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:43.439457    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xw2bp_3b40b006-204b-4291-901e-af512339c609/istio-proxy/1.log"
Jul 22 03:08:43.494143 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:43.494113    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-vdmlh_84cfc958-d425-471a-ab6f-cfcd0d8bc121/payload-processing/0.log"
Jul 22 03:08:44.693720 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:44.693686    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4ck5mw_aaae1329-63c3-460e-afcd-114656c1f9e1/istio-proxy/0.log"
Jul 22 03:08:44.915202 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:44.915175    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79676bf5cc-fgwxm_5d7f7d87-3443-42f1-b8c6-c07d6f272922/kube-auth-proxy/0.log"
Jul 22 03:08:44.952555 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:44.952530    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xw2bp_3b40b006-204b-4291-901e-af512339c609/istio-proxy/0.log"
Jul 22 03:08:44.980643 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:44.980606    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-xw2bp_3b40b006-204b-4291-901e-af512339c609/istio-proxy/1.log"
Jul 22 03:08:45.038783 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:45.038714    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-vdmlh_84cfc958-d425-471a-ab6f-cfcd0d8bc121/payload-processing/0.log"
Jul 22 03:08:48.135758 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:48.135735    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-xrb8s_8e539c5a-a01e-4319-9135-774e86dc565f/migrator/0.log"
Jul 22 03:08:48.157777 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:48.157744    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-xrb8s_8e539c5a-a01e-4319-9135-774e86dc565f/graceful-termination/0.log"
Jul 22 03:08:49.751188 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:49.751157    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pnw2f_32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b/kube-multus-additional-cni-plugins/0.log"
Jul 22 03:08:49.774938 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:49.774915    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pnw2f_32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b/egress-router-binary-copy/0.log"
Jul 22 03:08:49.799006 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:49.798988    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pnw2f_32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b/cni-plugins/0.log"
Jul 22 03:08:49.822580 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:49.822562    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pnw2f_32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b/bond-cni-plugin/0.log"
Jul 22 03:08:49.847013 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:49.846960    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pnw2f_32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b/routeoverride-cni/0.log"
Jul 22 03:08:49.869821 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:49.869801    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pnw2f_32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b/whereabouts-cni-bincopy/0.log"
Jul 22 03:08:49.892368 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:49.892352    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pnw2f_32559c29-8fd8-4c7b-8bdc-aa9dc7607f3b/whereabouts-cni/0.log"
Jul 22 03:08:50.109068 ip-10-0-140-200 kubenswrapper[2565]: I0722 03:08:50.109027    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-kvk2x_e736b988-93b9-40f9-9ab3-e3ee1a72f5a8/kube-multus/0.log"